Harmony in otion Zohar Barzelay Yoav Y Schechner

Harmony in♫ ♫otion Zohar Barzelay , Yoav Y. Schechner Dept. Elect. Eng. Technion – Israel Institute of Technology Ack: Einav Namer, Yael Waissman, ISF 1

“Harmony in ♫♫otion” Barzelay, Schechner Violin-guitar: raw 2

“Harmony in ♫♫otion” Barzelay, Schechner Violin: Detected and Recovered 3

“Harmony in ♫♫otion” Barzelay, Schechner Guitar: Detected and Recovered 4

5 Video features: track all Find the best Barzelay & Schechner, Harmony in Motion

6 Finding an Audio-Visual Object (AVO) Barzelay & Schechner, Harmony in Motion

7 Spatial matching: Many “coincidences” ? ? Corresponding images? * Always: unmatched features * Good image match: many “coincidences” * Spatial Edges ? Barzelay & Schechner, Harmony in Motion

8 Spatial matching Audio-Visual matching * Feature-based * Feature = significant change in space: edge, corner * Feature = significant change in time: temporal-edge * Maximize coincidences * No need to match everything Barzelay & Schechner, Harmony in Motion

Feature-based Cross-Modal Matching Barzelay & Schechner, Harmony in Motion 9

Feature-based Cross-Modal Matching Barzelay & Schechner, Harmony in Motion 9

Feature-based Cross-Modal Matching Acceleration time [frames] Barzelay & Schechner, Harmony in Motion 10

Feature-based Cross-Modal Matching Amplitude t ‘Visual Onsets’ ‘Audio Onsets’ 1 1 0 0 t t 11

Audio-Visual Coincidences Barzelay & Schechner, Harmony in Motion 12

13 Audio Pre-processing amplitude 0 energy t F 0 frequency Spectrogram frequency 0 Barzelay & Schechner, Harmony in Motion t

14 Significant change in audio spectrogram temporal derivative frequency 0 t Beginning of new sounds Audio Onsets 0 Barzelay & Schechner, Harmony in Motion t

15 Handling pitch-drift Barzelay & Schechner, Harmony in Motion

16 Handling pitch-drift spectrogram non-directional spectrogram derivative Barzelay & Schechner, Harmony in Motion

Visual Matching 17 1 0 t t

Visual Matching 18 0 1 -4 0 t Amplitude 1 1 0 -5 t t

Ranking Criterion 19 1 0 t t 0 1 coincidences 0 t Barzelay & Schechner, Harmony in Motion inconsistencies

Residual Audio Onsets 20 1 0 Residual Onsets t 0 1 coincidences 0 t Barzelay & Schechner, Harmony in Motion

Sequential Object Detection 21 Residual Onsets 1 0 t Amplitude 1 0 t Barzelay & Schechner, Harmony in Motion

“Harmony in ♫♫otion” Barzelay, Schechner Speech: raw 22

“Harmony in ♫♫otion” Barzelay, Schechner Speech A-B-C: Detected & Recovered 23

“Harmony in ♫♫otion” Barzelay, Schechner Speech 1 -2 -3: Detected & Recovered 24

25 Audio Isolation

26 Audio Pre-processing amplitude 0 energy t F 0 frequency Spectrogram frequency 0 Barzelay & Schechner, Harmony in Motion t

Audio Isolation 27 Corresponding Onsets t Spectrogram frequency 0 Barzelay & Schechner, Harmony in Motion t

Audio Isolation 27 Corresponding Onsets t Spectrogram c i n o m Har frequency s d n u o S 0 t

28 Fourier representation phase amplitude 0 t 0 frequency F Sp ec trog energy ram frequency 0 Barzelay & Schechner, Harmony in Motion t

29 Filtered audio old phase amplitude 0 0 frequency F t -1 Sp ec trog energy ram frequency 0 Barzelay & Schechner, Harmony in Motion t

Limitations: Temporal Tolerance 00: 16 ¼ sec 0 t t t 1 0 t Barzelay & Schechner, Harmony in Motion 30

Limitations: Audio Sparsity Overlapping audio onsets Sounds may overlap in time Onsets should not 31 Time-Frequency overlap frequency t Barzelay & Schechner, Harmony in Motion

Detection Parameters 32 00: 15 Feature-Detection: –edge scale –significance level –pruning Visual Edges: acceleration 1 0 t time Barzelay & Schechner, Harmony in Motion

“Harmony in ♫♫otion” Barzelay, Schechner Dual Viloin 33

“Harmony in ♫♫otion” Barzelay, Schechner 34

“Harmony in ♫♫otion” Barzelay, Schechner 35

Feature-based Cross-Modal Association • Features: Temporal Audio/Visual Edges. • Simultaneous Objects + Sounds. • A General Concept. 36