DSPCIS PartIII Optimal Adaptive Filters Chapter7 Wiener Filters

DSP-CIS Part-III : Optimal & Adaptive Filters Chapter-7 : Wiener Filters and the LMS Algorithm Marc Moonen Dept. E. E. /ESAT-STADIUS, KU Leuven marc. moonen@esat. kuleuven. be www. esat. kuleuven. be/stadius/

Part-III : Optimal & Adaptive Filters Chapter-7 Wieners Filters & the LMS Algorithm • • Introduction / General Set-Up Applications Optimal Filtering: Wiener Filters Adaptive Filtering: LMS Algorithm Chapter-8 – Recursive Least Squares Algorithms • • Least Squares Estimation Recursive Least Squares (RLS) Square Root Algorithms Fast RLS Algorithms – Kalman Filters Chapter-9 • Introduction – Least Squares Parameter Estimation • Standard Kalman Filter • Square-Root Kalman Filter DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 2 / 40

1. ‘Classical’ Filter Design See Part-II 2. ‘Optimal’ Filter Design realizations of DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm Norbert Wiener (1894 -1964) Introduction / General Set-Up 3 / 40

Introduction / General Set-Up DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 4 / 40

Introduction / General Set-Up 3. ‘Adaptive’ Filters DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 5 / 40

Introduction / General Set-Up DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 6 / 40

Applications ‘plant’ can be any system Optimal/adaptive filter provides mathematical model for input/outputbehavior of the plant DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 7 / 40

Applications echo path near-end signal + echo DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 8 / 40

Applications to/from network ‘Hybrid’ is never ideally matched to line impedance, hence generates echo of transmitted signal into received signal DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 9 / 40

Applications noise signal + noise DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 10 / 40

Applications noise signal + noise DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 11 / 40

Applications DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 12 / 40

Applications DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 13 / 40

Applications DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 14 / 40

Applications DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 15 / 40

Optimal Filtering : Wiener Filters Have to decide on 2 things. . 1 2 DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 16 / 40

1 u[k] e[k] DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm y[k] d[k] PS: Shorthand notation uk=u[k], yk=y[k], dk=d[k], ek=e[k], Filter coefficients (‘weights’) are wl (replacing bl of previous chapters) For adaptive filters wl also have a time index wl[k] Optimal Filtering : Wiener Filters 17 / 40

Optimal Filtering : Wiener Filters DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 18 / 40

Optimal Filtering : Wiener Filters PS: Can generalize FIR filter to ‘multi-channel FIR filter’ example: see page 11 DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 19 / 40

Optimal Filtering : Wiener Filters PS: Special case of ‘multi-channel FIR filter’ is ‘linear combiner’ FIR filter may then also be viewed as special case of ‘linear combiner’ where input signals are delayed versions of each other DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 20 / 40

Optimal Filtering : Wiener Filters 2 DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 21 / 40

Optimal Filtering : Wiener Filters MMSE cost function can be expanded as… DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 22 / 40

Optimal Filtering : Wiener Filters Correlation matrix has a special structure… DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 23 / 40

Optimal Filtering : Wiener Filters MMSE cost function can be expanded as…(continued) This is the ‘Wiener Filter’ solution DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 24 / 40

Optimal Filtering : Wiener Filters How do we solve the Wiener–Hopf equations? ( L+1 linear equations in L+1 unknowns) O(L 3) O(L 2) = used intensively in applications, e. g. in speech codecs, etc. details omitted, see Appendix DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 25 / 40

Adaptive Filtering: LMS Algorithm How do we solve the Wiener–Hopf equations? Alternatively, an iterative steepest descent algorithm can be used This will be the basis for the derivation of the Least Mean Squares (LMS) adaptive filtering algorithm… Bernard Widrow 1965 (https: //www. youtube. com/watch? v=hc 2 Zj 55 j 1 z. U) DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 26 / 40

Adaptive Filtering: LMS Algorithm How do we compute the Wiener filter? 1) Cfr supra: By solving Wiener-Hopf equations (L+1 equations in L+1 unknowns) 2) Can also apply iterative procedure to minimize MMSE criterion, e. g. here n is iteration index μ is ‘stepsize’ (to be tuned. . ) DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 27 / 40

Adaptive Filtering: LMS Algorithm Bound on stepsize ? DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 28 / 40

Adaptive Filtering: LMS Algorithm Convergence speed? è small λ_i implies slow convergence DSP-CIS 2016 / <<λ_max Part-III / Chapter-7: Wiener Filtersμ) & the LMS Algorithm è λ_min (hence small implies *very* slow convergence 29 / 40

Adaptive Filtering: LMS Algorithm as follows Replace n+1 by n for convenience… Then replace iteration index n by time index k k (i. e. perform 1 iteration per sampling interval) Then leave out expectation operators (i. e. replace expected values by instantaneous estimates) DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 30 / 40

Adaptive Filtering: LMS Algorithm Simple algorithm, can even draw signal flow graph (=realization)… DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 31 / 40

Adaptive Filtering: LMS Algorithm Whenever LMS has reached the WF solution, the expected value of (=estimated gradient in update formula) is zero, but the instantaneous value is generally non. DSP-CIS 2016 / Part-III / Chapter-7: Filters &move the LMS Algorithm 32 / 40 zero (=noisy), and hence LMSWiener will again away from the WF solution!

Adaptive Filtering: LMS Algorithm L L means step size has to be much smaller…! DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 33 / 40

Adaptive Filtering: LMS Algorithm LMS is an extremely popular algorithm many LMS-variants have been developed (cheaper/faster/…)… (see p. 35) K is block index, LB is block size DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 34 / 40

Adaptive Filtering: LMS Algorithm = LMS with normalized step size (mostly used in practice) DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 35 / 40

Appendix p i k s i h t e d i l s S DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 36 / 40

Appendix p i k s i h t e d i l s S DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 37 / 40

Appendix p i k s i h t e d i l s S DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 38 / 40

Appendix p i k s i h t e d i l s S DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 39 / 40

Appendix p i k s i h t e d i l s S DSP-CIS 2016 / Part-III / Chapter-7: Wiener Filters & the LMS Algorithm 40 / 40