Music Notation Music Representation AND Intelligence Donald Byrd

Music Notation, Music Representation, AND Intelligence Donald Byrd School of Music, Indiana University 3 February 2005 minor rev. 5 February 1

Overview • Music representations: from abstract to concrete (notation) • Try to assume just the right things: e. g. , no knowledge of music or notation • For motivation, focus on real-world content-based music-IR situations • Organization of the Talk I. Motivation: Why is this Important and/or Interesting? II. Representation and Semantics III. Music Notation, Representation, and Intelligence IV. Conclusions 2

You Are Here I. Motivation: Why is this Important and/or Interesting? II. Representation and Semantics III. Music Notation, Representation, and Intelligence IV. Conclusions 3

Audio-to-Audio Music “Retrieval” • “Shazam - just hit 2580 on your mobile phone and identify music” • Query: • Match: • Fantastically impressive to many people • Have they solved all the problems of music IR? No, (almost) none! • Reason: intended signal & match are identical => no time warping, let alone higher-level problems (perception/cognition) 4

Similarity Scale for Content-Based Music IR Relationship categories describing what’s in common between items whose similarity is to be evaluated (from closest to most distant) • For material in notation form, distinctions among (1), (2), and (3) don’t apply: it’s just “Same music, arrangement” 1. Same music, arrangement, performance, & recording (Shazam) 2. Same music, arrangement, performance; different recording 3. Same music, arrangement; different performance, recording 4. Same music, different arrangement; or different but closely-related music, e. g. , simpler variations (Mozart, etc. ), minor revs. (OMRAS, etc. ) 5. Different & less closely-related music: freer variations (Schumann, etc. ), extensive revisions (AI) 6. Music in same genre, etc. (AI? ) 7. Music influenced by other music (AI!) 5

OMRAS Polyphonic Audio Music IR: A Task that Needs Note Representation • Started with recordings of Bach preludes and fugues • Did polyphonic (several notes at once) music recognition • Polyphonic audio -> events is an open research problem • Converted results to MIDI, used as queries against database of c. 3000 pieces in MIDI form • One of worst-sounding cases: Prelude in G Major from Well. Tempered Clavier, Book I • Outcome: the actual piece was ranked 1 st! • Models built from notation database, but note data only • Query (audio -> MIDI -> audio) • Match (original audio recording) 6

Basic Representations of Music & Audio (e. g. , CD, MP 3): like speech Time-stamped Events (e. g. , MIDI file): like unformatted text Music Notation (sheet music): like HTML text 7

Basic Representations of Music & Audio • MIDI = Musical Instrument Digital Interface: simple, very standard low-bandwidth protocol (from early 1980’s) Audio Time-stamped Events Common examples CD, MP 3 file Unit Sample Explicit structure Event Music Notation Standard MIDI File Sheet music Note, clef, lyric, etc. none little (partial voicing much (complete information) voicing information) • Converting to form with less explicit structure (to left): moderately difficult • Converting to form with more explicit structure (to right): very difficult 8

Music-IR Problems that Needs More Structure • Joan Public’s problem: find a song, given some of the melody and some lyrics – Needs notes and text (lyrics) – Common question for music librarians, esp. in public libraries • Musicologist’s problem: authorship/origin of works in manuscripts – Full symbolic data is important, even “insignificant” details of notation (John Howard) 9

You Are Here I. Motivation: Why is this Important and/or Interesting? II. Representation and Semantics III. Music Notation, Representation, and Intelligence IV. Conclusions 10

Representation, from Abstract to Concrete • Cf. Basic Representations of Music & Audio • Abstract: represention: semantics only • Intermediate: syntax (mapping rules)? • Concrete – for use by computers: encoding – for use by humans: if visual, notation (involves graphics and/or typography) • Analogous to knowledge representation vs. data structure 11

Semantics in Music • Denotation (explicit, well-defined). . . • vs. Connotation (implicit, ill-defined) • In text – Two “definitions” of pig: • 1. Ugh! Dirty, evil-smelling creatures, wallowing in filthy sties! (Hayakawa) • 2. Mammal with short legs, cloven hoofs, bristly hair, and a cartilaginous snout used for digging (Amer. Heritage) – Prose is “mostly” denotation – Poetry is art => connotation much more important • Music is always art, & only connotation! • Major issue for content-based music IR 12

From Representation to Notation • Choosing a representation inevitably introduces bias • Given a representation, choosing notation inevitably introduces more bias • Important to consider the purpose (R. Davis et al; Wiggins et al) • For huge body of important music, we have no choice: notation is CMN (Conventional Music Notation)! – – Really “CWMN” (W = Western) Alternative for some music: tablature (guitar, lute, etc. ) CMN is among the most successful notations ever. . . but enormously complex and subtle 13

Notation Says Much about Representation • CMN standard for Western music after c. 1650 • Evolved for “classical” music, but heavily used for very wide range (pop, jazz, folk, etc. ) • Composers/arrangers/transcribers have pushed it hard => reveals things about music representation in general • Will concentrate on notation (CMN) 14

You Are Here I. Motivation: Why is this Important and/or Interesting? II. Representation and Semantics III. Music Notation, Representation, and Intelligence IV. Conclusions 15

How to Read Music (CMN) Without Really Trying: The Basics • Four basic parameters of a musical note 1. Pitch: how high or low sound is 2. Duration: how long the note lasts 3. Loudness: perceptual analog of amplitude 4. Timbre or tone quality • Above in decreasing order of importance for most Western music • Principles of CMN (& e 1) 1. Pitch on vertical axis: clef gives offset (“zero”) 2 a. Duration indicated by note/rest shapes 2 b. Start times (sum of durations in the voice) on horizontal axis 3. Loudness indicated by signs like p , mf , etc. 4. Timbre indicated with words like “violin”, “horn”, “pizzicato” 16

Why is Musical Information Hard to Handle? 1. Units of meaning: not clear anything in music is analogous to words (all representations) 2. Polyphony: “parallel” independent voices, something like characters in a play (all representations) 3. Recognizing notes (audio only) 4. Other reasons 17

Units of Meaning (Problem 1) • Not clear anything in music is analogous to words – No explicit delimiters (like Chinese) – Experts don’t agree on “word” boundaries (unlike Chinese) • • • Are notes like words? No. Relative, not absolute, pitch is important Are pitch intervals like words? No. They’re too low level: more like characters Are pitch-interval sequences like words? In some ways, but – Ignores note durations – Ignores relationships between voices (harmony) – Probably little correlation with semantics 18

Independent Voices in Music (Problem 2) (& e 2) J. S. Bach: “St. Anne” Fugue, beginning 19

Independent Voices in Text MARLENE. What I fancy is a rare steak. Gret? ISABELLA. I am of course a member of the / Church of England. * GRET. Potatoes. MARLENE. *I haven’t been to church for years. / I like Christmas carols. ISABELLA. Good works matter more than church attendance. --Caryl Churchill: “Top Girls” (1982), Act 1, Scene 1 Performance (time goes from left to right): M: What I fancy is a rare steak. Gret? I: G: I haven’t been. . . I am of course a member of the Church of England. Potatoes. 20

Complex Notation: Multiple Voices (& e 3) Multiple voices on a staff rapidly gets worse with more than 2 (Telemann “Liebe, Liebe”): • 2 voices in mm. 5 -6: not bad: stem direction is enough • 3 voices in m. 7: notes must move sideways • 4 voices in m. 8: almost unreadable—without color! • Still acceptable because specific voice is rarely important 21

Problems: Example 1 (superficial but interesting) • • Ravel work has slur with 7 inflection points Impressive, but complexity is purely graphical No big deal in terms of representation …but influence of performance on notation is revealing 22

Duration and Higher-Level Concepts of Time • Schubert Impromptu (& e 4) • Measures: everything between barlines • Time signature: 3/4 = 3 quarter notes per measure • Triplets: 3 notes in the time normally used by 2 – General concept is tuplets 23

Problems: Example 2 (Deep) • Chopin Nocturne has nasty situation (& e 5) • One notehead is triplet in one voice, but normal duration in another • “Semantics” (execution) well-defined, obvious – Note starts 1/16 before barline… – But also (2/3)*(1/16) before barline! How to play? • Reason: musical necessity • Solution for performer: “rubato” • Solution for music IR program: ? 24

Problems: Example 3 (Medium) • Bach: time signature change in middle of measure • (& e 6) • “Semantics” well-defined and obvious – Measure has duration of 18 16 ths… – But not until the middle of the measure! • How does this make sense? • Triplets express same relationship as equivalent simple/compound meter • Invisible (unmarked) triplets • Cf. Bach Prelude: two time signatures at once (& • Reason: avoid clutter e 7) 25

Problem 4 (Medium) • • • Brahms Capriccio (& e 8) Time signature 6/8 => measure lasts 12 16 ths A dotted half note always lasts 12 16 ths… but here it clearly lasts only 11 16 ths! Reason: avoid clutter 26

Two Ways to Have Two Clefs at Once • Clef gives vertical offset to determine pitch • Debussy (& e 9) – Bizarrely obvious something odd involving clefs • Ravel (& e 10) – Only comparing time signature (3/8) and note durations makes it clear both clefs affect whole measure • Reason: save space (by avoiding a 3 rd staff) 27

Surprise: Music Notation has Meta. Principles! (1) 1. Maximize readability (intelligibility) – – Avoid clutter = “Omit Needless Symbols” Try to assume just the right things for audience Audience for CMN is (primarily) performers General principle of any communication • Applies to talks as well as music notation! – Examples: Schubert, Bach, Brahms 28

Surprise: Music Notation has Meta. Principles! (2) 2. Minimize space used – Save space => fewer page turns (helps performer); also cheaper to print (helps publisher) – Squeezing much music into little space is a major factor in complexity of CMN – Especially important for music: real-time, hands full – Examples: Telemann, Debussy, Ravel 29

The “Rules” of Music Notation • Tempting to assume that rules of such an elaborate & successful system as CMN work (self-consistent, reasonably unambiguous, etc. ) in every case • But (a) “rules” evolved, with no established authority; (b) many of the “rules” are very nebulous • In common cases, there's no problem • If you try to make every rule as precise as possible, result is certainly not self-consistent • Trying to save space makes rules interact; something has to give! 30

Music Notation Software and Intelligence • Despite odd notation, really nothing strange going on in almost all of these examples – Ravel slur, Debussy & Ravel 2 simultaneous clefs, Bach & Schubert invisible triplets, Brahms “short” dotted-half note, Telemann 4 voices/staff are all simple situations – Chopin Nocturne is complex • Programmers try to help users by having programs do things “automatically” • A good idea if software knows enough to do the right thing “almost all” the time—but no program does! • Notation programs convert CMN to performance (MIDI) and vice-versa => requires shallow “semantics”; makes things much harder 31

You Are Here I. Motivation: Why is this Important and/or Interesting? II. Representation and Semantics III. Music Notation, Representation, and Intelligence IV. Conclusions 32

Conclusions: Review (1) • • Representations express Semantics of Music; Denotation & Connotation Principles of CMN Meta-Principles of CMN 1. Maximize readability; Omit Needless Symbols • Try to assume just the right things for audience • General principle of any communication 2. Minimize space used • Save space => fewer page turns, less paper 33

Conclusions: Review (2) • We need CMN or equivalent to solve spectrum of music-IR (and other music-IT) problems – – – But CMN can’t represent everything we want Even when it can, may not, at least explicitly Need high-level intelligence to interpret Solution: unknown Likely to require major funding : -) 34

Conclusions: Why is this really Important and/or Interesting? • Some problems directly related to other areas of informatics – Example: Approximate string matching in bioinformatics • Encourages progress on real semantics – Connotation is an important part of meaning in everything – Can often ignore, but any semantics in arts forces you to deal with connotation – Music is at least as quantifiable as any art, so likely to be more tractable than others! 35

You Are Here The End 36