Speech Audio Processing PartII Digital Audio Signal Processing

Speech & Audio Processing - Part–II Digital Audio Signal Processing Marc Moonen Dept. E. E. /ESAT-STADIUS, KU Leuven marc. moonen@esat. kuleuven. be homes. esat. kuleuven. be/~moonen/

Speech & Audio Processing • Part-I (H. Van hamme) speech recognition speech coding (+audio coding) speech synthesis (TTS) • Part-II (M. Moonen): Digital Audio Signal Processing microphone array processing noise cancellation acoustic echo cancellation acoustic feedback- cancellation active noise control 3 D audio PS: selection of topics Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 2

Digital Audio Signal Processing • • Aims/scope Case study: Hearing instruments Overview Prerequisites Lectures/course material/literature Exercise sessions/project Exam Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 3

Aims/Scope Aim is 2 -fold : • Speech & audio per se S & A industry in Belgium/Europe/… • Basic signal processing theory/principles : Optimal filters Adaptive filter algorithms (Filtered-X LMS, . . ) Kalman filters etc. . . Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 4

Case Study: Hearing Instruments 1/14 Hearing © www. cm. be • Outer ear/middle ear/inner ear • Tonotopy of inner ear: spatial arrangement of where sounds of different frequency are processed Low-freq tone = Cochlea Neural activity for low-freq tone High-freq tone Digital Audio Signal Processing: Introduction Version 2015 -2016 Neural activitity for high-freq Lecture-1: Introduction tone p. 5

Case Study: Hearing Instruments 2/14 Hearing loss types: • conductive • sensorineural • mixed One in six adults (Europe) …and still increasing Typical causes: • aging • exposure to loud sounds • … Digital Audio Signal Processing: Introduction Version 2015 -2016 [Source: Lapperre] Lecture-1: Introduction p. 6

Case Study: Hearing Instruments 3/14 Hearing impairment : Dynamic range & audibility Normal hearing subjects Hearing impaired subjects Level 100 d. B Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 7

Case Study: Hearing Instruments 4/14 Hearing impairment : Dynamic range & audibility Dynamic range compression (DRC) (…rather than `amplification’) 100 d. B Output Level (d. B) Level 100 d. B 100 d. B Input Level (d. B) Design: multiband DRC, attack time, release time, … Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 8

Case Study: Hearing Instruments 5/14 Hearing impairment : Audibility vs speech intelligibility • Audibility does not imply SNR intelligibility 20 d. B • Hearing impaired subjects need 5. . 10 d. B larger signal -to-noise ratio (SNR) for 0 d. B speech understanding in noisy 30 50 70 90 environments Hearing loss (d. B, 3 -freq-average) • Need for noise reduction (=speech enhancement) algorithms: • State-of-the-art: monaural 2 -microphone adaptive noise reduction • Near future: binaural noise reduction (see below) • Not-so-near future: multi-node noise reduction (see below) Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 9

Hearing Aids (HAs) • Audio input/audio output (`microphone-processing-loudspeaker’) • ‘Amplifier’, but so much more than an amplifier!! • History: Horns/trumpets/… `Desktop’ HAs (1900) Wearable HAs (1930) Digital HAs (1980) 2007 (Oticon) • • 1921 Case Study: Hearing Instruments 6/14 • State-of-the-art: • MHz’s clock speed • Millions of arithmetic operations/sec, … • Multiple microphones Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 10

• Audio input/electrode stimulation output • Stimulation strategy + preprocessing similar to HAs • History: Intra-cochlear • • Volta’s experiment… First implants (1960) Commercial CIs (1970 -1980) Digital CIs (1980) electrode • State-of-the-art: • MHz’s clock speed, Mops/sec, … • Multiple microphones © Cochlear Ltd Cochlear Implants (CIs) Alessandro Volta 1745 -1827 Case Study: Hearing Instruments 7/14 Other: Bone anchored HAs, middle ear implants, … Digital Audio Signal Processing: Introduction Electrical stimulation p. 11 Version 2015 -2016 Lecture-1: Introduction for low frequency for high frequency

Case Study: Hearing Instruments 8/14 © Cochlear Ltd • External Processor Digital/analog-conversion Digital processing & filterbank Etc. . • Coil Inductive/magnetic coupling • Implant Electrode array PS: number of CI-implantees worldwide approx. 200. 000 PS: 1 CI is approx. 25 k. EURO, plus surgery, revalidation, . . PS: 3 companies (Cochlear Lt. D, Med-El, Advanced Bionics) Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 12

Case Study: Hearing Instruments 9/14 Technology challenges in hearing instruments • Small form factor (cfr. user acceptance) • Low power: 1… 5 m. W (cfr. battery lifetime ≈ 1 week) • Low processing delay: 10 msec (cfr. synchronization with lip reading) DSP challenges in hearing instruments • • Dynamic range compression (cfr supra) Dereverberation: undo filtering (`echo-ing’) by room acoustics Feedback cancellation Noise reduction Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 13

Case Study: Hearing Instruments 10/14 DSP Challenges: Feedback Cancellation • Problem statement: Loudspeaker signal is fed back into microphone, then amplified and played back again • Closed loop system may become unstable (howling) • Similar to feedback problem in public address systems (for the musicians amongst you) Model F Digital Audio Signal Processing: Introduction Similar to echo cancellation in GSM handsets, Skype, … but more difficult due to signal correlation Version 2015 -2016 Lecture-1: Introduction p. 14

Case Study: Hearing Instruments 11/14 DSP Challenges: Noise reduction Multimicrophone ‘beamforming’, typically with 2 microphones, e. g. ‘directional’ front microphone and ‘omnidirectional’ back microphone “filter-and-sum” the microphone signals Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 15

Case Study: Hearing Instruments 12/14 Binaural hearing: Binaural auditory cues • ITD (interaural time difference) • ILD (interaural level difference) signal ILD ITD • Binaural cues (ITD: f < 1500 Hz, ILD: f > 2000 Hz) used for • Sound localization • Noise reduction =`Binaural unmasking’ (‘cocktail party’ effect) 0 -5 d. B Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 16

Case Study: Hearing Instruments 13/14 Binaural hearing aids • Two hearing aids (L&R) with wireless link & cooperation • Opportunities: • More signals (e. g. 2*2 microphones) • Better sensor spacing (17 cm i. o. 1 cm) • Constraints: power/bandwith/delay of wireless link • . . 10 k. Bit/s: coordinate program settings, parameters, … • . . 300 k. Bits/s: exchange 1 or more (compressed) audio signals • Challenges: • Improved localization through cue preservation • Improved noise reduction + benefit from binaural unmasking • Signal selection/filtering, audio coding, synchronisation, … Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 17

Case Study: Hearing Instruments 14/14 Future: Multi-node noise reduction – sensor networks Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 18

Overview : Lecture-2 Microphone Array Processing Referred to as ‘spatial filtering’ (similar to ‘spectral filtering’) or ‘beamforming’ Fixed vs. adaptive beamforming Filter-and-sum beamformer Application: hearing aids Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 19

Overview : Lecture-3 Noise Reduction `microphone_signal[k] = speech[k] + noise[k]’ • Single-microphone noise reduction – Spectral Subtraction Methods (spectral filtering) – Iterative methods based on speech modeling (Wiener & Kalman Filters) • Multi-microphone noise reduction – Beamforming revisited – Optimal filtering approach : spectral+spatial filtering Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 20

Overview : Lecture-4 Guest Lecture Prof. Tom Francart, KU Leuven, Exp. ORL ‘Evaluation of Audio/Speech Signal Processing Algorithms’ – Speech intelligibility in noise – Instrumental meassures – Behavioral measures Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 21

Overview : Lecture-5 Adaptive Filters for Acoustic Echo- and Feedback Cancellation Adaptive filtering problem: • non-stationary/wideband/… speech signals • non-stationary/long/… acoustic channels Adaptive filtering algorithms AEC Control AEC Post-processing Stereo AEC Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 22

Overview : Lecture-5 Adaptive Filters for Acoustic Echo- en Feedback Cancellation (continued) • Hearing aids, public address (PA) systems • correlation between filter input (`x ’) and near-end signal (‘ n ’) • fixes : noise injection, pitch shifting, notch filtering, … amplifier Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 23

Overview : Lecture-6 Kalman Filters for Acoustic Echo- en Feedback Cancellation • ‘Generalizes’ Wiener Filter. . • . . based on model for time-evolution of filter coefficients amplifier Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 24

Overview : Lecture-7 Active Noise Control • Solution based on `filtered-X LMS’ • Application : active headsets/ear defenders Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 25

Overview : Lecture-7 3 D Audio & Loudspeaker Arrays Binaural synthesis …with headphones head related transfer functions (HRTF) …with 2+ loudspeakers (`sweet spot’) crosstalk cancellation Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 26

Overview : Lecture-8 Guest Lecture Dr. Enzo De Sena, KU Leuven, ESAT/STADIUS ‘Auralization for Architectural Acoustics, Virtual Reality and Computer Games - from Physical to Perceptual Rendering of Dynamic Sound Scenes’ Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 27

Aims/Scope (revisited) Aim is 2 -fold : • Speech & audio per se • Basic signal processing theory/principles : Optimal filtering / Kalman filters (linear/nonlinear) here : echo cancellation, speech enhancement other : automatic control, spectral estimation, . . . Advanced adaptive filter algorithms here : acoustic echo cancellation other : digital communications, . . . Filtered-X LMS here : 3 D audio other : active noise/vibration control Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 28

Lectures: 1 Intro + 7 Lectures PS: Time budget = 1*(2 hrs)*2 +7*(2 hrs)*4 = 60 hrs Course Material: Slides – Use version 2015 -2016 ! – Download from DASP webpage homes. esat. kuleuven. be/~dspuser/dasp/ Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 29

Prerequisites • H 197 Signals & Systems (JVDW) • HJ 09 Digital Signal Processing (I) (PW) signal transforms, sampling, multi-rate, DFT, … • HC 63 DSP-CIS (MM) filter design, filter banks, optimal & adaptive filters Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 30

Literature (general) (available in DSP-CIS library) • Simon Haykin `Adaptive Filter Theory’ (Prentice Hall 1996) • P. P. Vaidyanathan `Multirate Systems and Filter Banks’ (Prentice Hall 1993) Literature (specialized) (available in DSP-CIS library) • S. L. Gay & J. Benesty `Acoustic Signal Processing for Telecommunication’ (Kluwer 2000) • M. Kahrs & K. Brandenburg (Eds) `Applications of Digital Signal Processing to Audio and Acoustics’ (Kluwer 1998) • B. Gold & N. Morgan `Speech and Audio Signal Processing’ (Wiley 2000) Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 31

Exercise Sessions/Project Direction-of-arrival θ Acoustic source localization – – Direction-of-arrival estimation Noise reduction Synthesis Simulated set-up Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 32

Acoustic Source Localization Project PS: groups of 2 • Runs over 4 weeks (non-consecutive) • Each week – 1 PC/Matlab session (supervised, 2. 5 hrs) – 2 ‘Homework’ sesions (unsupervised, 2*2. 5 hrs) PS: Time budget = 4*(2. 5 hrs+5 hrs) = 30 hrs • ‘Deliverables’ after week 2 & 4 • Grading: based on deliverables, evaluated during sessions • TAs: guiliano. bernardi@esat (English+Italian) Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 33

Acoustic Source Localization Project Work Plan – Week 1: Matlab acoustic simulation environment – Week 2: Direction-of-arrival (Do. A) estimation based on the ‘MUSIC’ algorithm *deliverable* – Week 3: Do. A estimation + noise reduction (‘DOA informed beamforming’) – Week 4: Binaural synthesis and 3 D audio *deliverable* . . be there ! Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 34

Exam • Oral exam, with preparation time • Open book • Grading 7 for question-1 7 for question-2 +6 for project ___ = 20 Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 35

September Retake Exam • Oral exam, with preparation time • Open book • Grading 7 for question-1 7 for question-2 +6 for question-3 ___ = 20 Digital Audio Signal Processing: Introduction (related to project work) Version 2015 -2016 Lecture-1: Introduction p. 36

Website 1) TOLEDO 1) http: //homes. esat. kuleuven. be/~dspuser/dasp/ • • • Contact: guiliano. bernardi@esat Slides (use `version 2015 -2016’ !!) Schedule DSP-library FAQs (send questions to marc. moonen@esat) Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 37

Questions? 1) Ask teaching assistant (during exercises sessions) 2) E-mail questions to teaching assistant or marc. moonen@esat 3) Make appointment marc. moonen@esat ESAT Room B. 00. 14 Digital Audio Signal Processing: Introduction Version 2015 -2016 Lecture-1: Introduction p. 38