Speech Audio Processing PartII Digital Audio Signal Processing

Speech & Audio Processing - Part–II Digital Audio Signal Processing Marc Moonen Dept. E. E. /ESAT-STADIUS, KU Leuven marc. moonen@esat. kuleuven. be homes. esat. kuleuven. be/~moonen/

Speech & Audio Processing • Part-I (H. Van hamme) speech recognition speech coding (+audio coding) speech synthesis (TTS) • Part-II (M. Moonen): Digital Audio Signal Processing microphone array processing noise cancellation acoustic echo cancellation acoustic feedback- cancellation active noise control 3 D audio PS: selection of topics Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 2

Digital Audio Signal Processing • • Aims/scope Case study: Hearing instruments Overview Prerequisites Lectures/course material/literature Exercise sessions/project Exam Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 3

Aims/Scope Aim is 2 -fold : • Speech & audio per se S & A industry in Belgium/Europe/… • Basic signal processing theory/principles : Optimal filters Adaptive filter algorithms (APA, Filtered-X LMS, . . ) Kalman filters (linear/nonlinear) etc. . . Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 4

Case Study: Hearing Instruments 1/14 Hearing Low-freq tone © www. cm. be • Outer ear/middle ear/inner ear • Tonotopy of inner ear: spatial arrangement of where sounds of different frequency are processed = Cochlea High-freq tone Neural activity for low-freq tone Neural activitity for high-freq tone Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 5

Case Study: Hearing Instruments 2/14 Hearing loss types: • conductive • sensorineural • mixed One in six adults (Europe) …and still increasing Typical causes: • aging • exposure to loud sounds • … [Source: Lapperre] Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 6

Case Study: Hearing Instruments 3/14 Hearing impairment : Dynamic range & audibility Normal hearing Hearing impaired subjects subjects Level 100 d. B Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 7

Case Study: Hearing Instruments 4/14 Hearing impairment : Dynamic range & audibility Dynamic range compression (DRC) (…rather than `amplification’) 100 d. B Output Level (d. B) Level 100 d. B 100 d. B Input Level (d. B) Design: multiband DRC, attack time, release time, … Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 8

Case Study: Hearing Instruments 5/14 Hearing impairment : Audibility vs speech intelligibility • Audibility does not imply intelligibility SNR 20 d. B • Hearing impaired subjects need 5. . 10 d. B larger signal -to-noise ratio (SNR) for speech understanding in noisy 0 d. B environments 30 50 70 90 Hearing loss (d. B, 3 -freq-average) • Need for noise reduction (=speech enhancement) algorithms: • State-of-the-art: monaural 2 -microphone adaptive noise reduction • Near future: binaural noise reduction (see below) • Not-so-near future: multi-node noise reduction (see below) Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 9

• • Horns/trumpets/… `Desktop’ HAs (1900) Wearable HAs (1930) Digital HAs (1980) • State-of-the-art: 2007 (Oticon) Hearing Aids (HAs) • Audio input/audio output (`microphone-processing-loudspeaker’) • ‘Amplifier’, but so much more than an amplifier!! • History: 1921 Case Study: Hearing Instruments 6/14 • MHz’s clock speed • Millions of arithmetic operations/sec, … • Multiple microphones Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 10

• Audio input/electrode stimulation output • Stimulation strategy + preprocessing similar to HAs • History: Intra-cochlear • • Volta’s experiment… First implants (1960) Commercial CIs (1970 -1980) Digital CIs (1980) electrode • State-of-the-art: • MHz’s clock speed, Mops/sec, … • Multiple microphones © Cochlear Ltd Cochlear Implants (CIs) Alessandro Volta 1745 -1827 Case Study: Hearing Instruments 7/14 Other: Bone anchored HAs, middle ear implants, … Electrical stimulation p. 11 Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction for low frequency for high frequency

Case Study: Hearing Instruments 8/14 Digital/analog-conversion (cfr infra) Digital processing & filterbank Etc. . • Coil © Cochlear Ltd • External Processor Inductive/magnetic coupling • Implant Electrode array PS: number of CI-implantees worldwide approx. 200. 000 PS: 1 CI is approx. 25 k. EURO, plus surgery, revalidation, . . PS: 3 companies (Cochlear Lt. D, Med-El, Advanced Bionics) p. 12 Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction 12/32

Case Study: Hearing Instruments 9/14 HA technology requirements • Small form factor (cfr. user acceptance) • Low power: 1… 5 m. W (cfr. battery lifetime ≈ 1 week) • Low processing delay: 10 msec (cfr. synchronization with lip reading) DSP challenges in hearing instruments • • Dynamic range compression (cfr supra) Dereverberation: undo filtering (`echo-ing’) by room acoustics Feedback cancellation Noise reduction Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 13

Case Study: Hearing Instruments 10/14 DSP Challenges: Feedback Cancellation • Problem statement: Loudspeaker signal is fed back into microphone, then amplified and played back again • Closed loop system may become unstable (howling) • Similar to feedback problem in public address systems (for the musicians amongst you) Model F - Similar to echo cancellation in GSM handsets, Skype, … but more difficult due to signal correlation Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 14

Case Study: Hearing Instruments 11/14 DSP Challenges: Noise reduction Multimicrophone ‘beamforming’, typically with 2 microphones, e. g. ‘directional’ front microphone and ‘omnidirectional’ back microphone “filter-and-sum” the microphone signals Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 15

Case Study: Hearing Instruments 12/14 Binaural hearing: Binaural auditory cues • ITD (interaural time difference) • ILD (interaural level difference) signal ILD ITD • Binaural cues (ITD: f < 1500 Hz, ILD: f > 2000 Hz) used for • Sound localization • Noise reduction =`Binaural unmasking’ (‘cocktail party’ effect) 0 -5 d. B Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 16

Case Study: Hearing Instruments 13/14 Binaural hearing aids • Two hearing aids (L&R) with wireless link & cooperation • Opportunities: • More signals (e. g. 2*2 microphones) • Better sensor spacing (17 cm i. o. 1 cm) • Constraints: power/bandwith/delay of wireless link • . . 10 k. Bit/s: coordinate program settings, parameters, … • . . 300 k. Bits/s: exchange 1 or more (compressed) audio signals • Challenges: • Improved localization through cue preservation • Improved noise reduction + benefit from binaural unmasking • Signal selection/filtering, audio coding, synchronisation, … Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 17

Case Study: Hearing Instruments 14/14 Future: Multi-node noise reduction – sensor networks Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 18

Overview General speech communication set-up : - background ‘noise’ noise suppression, source separation - far-end echoes acoustic echo cancellation - reverberation de-reverberation/deconvolution Applications : • teleconferencing/teleclassing • hands-free telephony • hearing aids, etc. . Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 19

Overview : Lecture-2&3 Microphone Array Processing Spatial filtering - Beamforming Fixed (Lecture 2) vs. adaptive (Lecture 3) Example filter-and-sum beamformer : Application: hearing aids Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 20

Overview : Lecture-4 Noise Reduction `microphone_signal[k] = speech[k] + noise[k]’ • Single-microphone noise reduction – Spectral Subtraction Methods (spectral filtering) – Iterative methods based on speech modeling (Wiener & Kalman Filters) • Multi-microphone noise reduction – Beamforming revisited – Optimal filtering approach : spectral+spatial filtering Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 21

Overview : Lecture-5 Acoustic Echo Cancellation Adaptive filtering problem: • non-stationary/wideband/… speech signals • non-stationary/long/… acoustic channels Adaptive filtering algorithms AEC Control AEC Post-processing Stereo AEC Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 22

Overview : Lecture-6 Acoustic Feedback Cancellation • • Ex: Hearing aids Ex: PA systems correlation between filter input (`x ’) and near-end signal (‘ n ’) fixes : noise injection, pitch shifting, notch filtering, . . . amplifier Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 23

Overview : Lecture-7 Active Noise Control • Solution based on `filtered-X LMS’ • Application : active headsets/ear defenders Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 24

Overview : Lecture-7 bis & 8 3 D Audio & Loudspeaker Arrays • Binaural synthesis …with headphones head related transfer functions (HRTF) …with 2+ loudspeakers (`sweet spot’) crosstalk cancellation Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 25

Aims/Scope (revisited) Aim is 2 -fold : • Speech & audio per se • Basic signal processing theory/principles : Optimal filtering / Kalman filters (linear/nonlinear) here : speech enhancement other : automatic control, spectral estimation, . . . Advanced adaptive filter algorithms here : acoustic echo cancellation other : digital communications, . . . Filtered-X LMS here : 3 D audio other : active noise/vibration control Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 26

Lectures: 7*2 hrs (Lectures 1 -7) + 1*1 hr (Lectures 8) – PS: Time budget = (15 hrs)*4 = 60 hrs Course Material: Slides – Use version 2013 -2014 ! – Download from DASP webpage http: //homes. esat. kuleuven. be/~dspuser/dasp/ Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 27

Prerequisites • H 197 Signals & Systems (JVDW) • HJ 09 Digital Signal Processing (I) (PW) signal transforms, sampling, multi-rate, DFT, … • HC 63 DSP-CIS (MM) filter design, filter banks, optimal & adaptive filters Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 28

Literature (General) (available in DSP-CIS library) • Simon Haykin `Adaptive Filter Theory’ (Prentice Hall 1996) • P. P. Vaidyanathan `Multirate Systems and Filter Banks’ (Prentice Hall 1993) Literature (specialized) (some available in DSP-CIS library) • S. L. Gay & J. Benesty `Acoustic Signal Processing for Telecommunication’ (Kluwer 2000) • M. Kahrs & K. Brandenburg (Eds) `Applications of Digital Signal Processing to Audio and Acoustics’ (Kluwer 1998) • B. Gold & N. Morgan `Speech and Audio Signal Processing’ (Wiley 2000) Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 29

Exercise Sessions/Project Direction-of-arrival θ Acoustic source localization – – Direction-of-arrival estimation Noise reduction Echo cancellation Simulated set-up Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 30

Acoustic Source Localization Project PS: groups of 2 • Runs over 4 weeks (non-consecutive) • Each week – 1 PC/Matlab session (supervised, 2. 5 hrs) – 2 ‘Homework’ sesions (unsupervised, 2*2. 5 hrs) PS: Time budget = 4*(2. 5 hrs+5 hrs) = 30 hrs • ‘Deliverables’ after week 2 & 4 • Grading: based on deliverables, evaluated during sessions • TAs: guiliano. bernardi@esat (English+Italian) Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 31

Acoustic Source Localization Project Work Plan – Week 1: Design Matlab simulation set-up – Week 2: Direction-of-arrival (Do. A) estimation *deliverable* – Week 3: Do. A estimation + noise reduction – Week 4: Do. A estimation + echo cancellation *deliverable* . . be there ! Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 32

Exam • Oral exam, with preparation time • Open book • Grading 7 for question-1 7 for question-2 +6 for project ___ = 20 Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 33

September Retake Exam • Oral exam, with preparation time • Open book • Grading 7 for question-1 7 for question-2 +6 for question-3 ___ = 20 (related to project work) Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 34

Website 1) TOLEDO 1) http: //homes. esat. kuleuven. be/~dspuser/dasp/ • • • Contact: guiliano. bernardi@esat Slides (use `version 2014 -2015’ !!) Schedule DSP-library FAQs (send questions to marc. moonen@esat) Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 35

Questions? 1) Ask teaching assistant (during exercises sessions) 2) E-mail questions to teaching assistant or marc. moonen@esat 3) Make appointment marc. moonen@esat ESAT Room 01. 69 Digital Audio Signal Processing: Introduction Version 2014 -2015 Lecture-1: Introduction p. 36