Open Issues in Constrained Blind Source Separation Jonathon

Open Issues in Constrained Blind Source Separation Jonathon Chambers Cardiff Professorial Research Fellow Cardiff School of Engineering Cardiff University, Wales, U. K. E-mail: chambersj@cf. ac. uk IEEE UKRI Talk, De Montfort Univ. , April 2, 04 1

Summary of Talk • Acknowledgement • Historical background & motivation • BSS with matrix constraints • Penalty functions in FD-BSS • Exploiting periodicity in BSS • Future application-driven challenges IEEE UKRI Talk, De Montfort Univ. , April 2, 04 2

Acknowledgements Jonathon Chambers wishes to express his sincere thanks for the support of Professor Andrzej Cichocki, Riken Brain Science Institute, Japan The invitation from the organising committee of the workshop to give this talk. His co-researchers: Drs Saeid Sanei, Maria Jafari and Wenwu Wang. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 3

LMS Algorithm B. Widrow, and M. E. Hoff, Jr. , “Adaptive switching circuits, ” IRE Wescon Conv. Rec. , pt. 4, pp. 96 -104, 1960. LMS Update IEEE UKRI Talk, De Montfort Univ. , April 2, 04 4

Historical Background • The field of conventional adaptive signal processing has been greatly enhanced by the exploitation of constrained optimisation • Constraints on the error, and/or structure or some norm of the weights via, for example, Lagrange multipliers and/or Karush-Khun-Tucker conditions IEEE UKRI Talk, De Montfort Univ. , April 2, 04 5

Historical Background Certain key papers: • O. L. Frost, III, “An algorithm for linearly constrained adaptive array processing, ” Proc. IEEE, Vol. 60(8), pp. 926 -925, 1972 • R. P. Gitlin et al. “The tap-leakage algorithm: an algorithm for the stable operation of a digitally implemented fractionally spaced equalizer, ” Bell Sys. Tech. Journal, Vol. 61(8), pp. 1817 -1839, 1982. • D. T. M. Slock, “Convergence behavior of the LMS and Normalised LMS Algorithms, ” IEEE Trans. Signal Processing, Vol. 41(9), pp. 2811 -2825, 1993. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 6

Historical Background Cont. • S. C. Douglas, “A family of normalized LMS algorithms, ” IEEE Signal Processing Letters, Vol. 1(3), pp. 49 -51, 1994. • S. C. Douglas, and M. Rupp, “A posteriori updates for adaptive filters, ” Asilomar Conference on Signals, Systems and Computers, Vol. 2, pp 1641 -1645, 1997. • T. Gänsler, et al. , “A robust proportionate affine projection algorithm for network echo cancellation, ” Proc. ICASSP 2000, Vol. 2, pp. 793 -796, 2000. • O. Vainia, “Polynomial constrained LMS adaptive algorithm for measurement signal processing, ” Proc. IECON 2002, Vol. 2, pp. 1479 -1482, 2002. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 7

Motivation “In many applications of Independent Component Analysis (ICA) and Blind Source Separation (BSS) estimated source signals and the mixing or separating matrices have some special structure or some constraints are imposed for the matrices…”, Cichocki and Georgiev, 2003 IEEE UKRI Talk, De Montfort Univ. , April 2, 04 8

Fundamental Model for Instantaneous Blind Source Separation Mixing Process s 1 s. N Unmixing Process x 1 H Unknown Y 1 x. M W YN Independent? Known IEEE UKRI Talk, De Montfort Univ. , April 2, 04 Adapt 9

Certain BSS Books • Andrzej Cichocki and Shun-Ichi Amari, Adaptive Blind Signal and Image Processing, Wiley, 2002 • Simon Haykin Unsupervised Adaptive Filtering, Vols. I and II, Wiley, 2000 • Aapo Hyvärinen, Juha Karhunen and Erkki Oja, Independent Component Analysis, Wiley, 2001 • Te-Won Lee, Independent component analysis: theory and applications, Kluwer, 1998 IEEE UKRI Talk, De Montfort Univ. , April 2, 04 10

BSS References • A. Mansour and M. Kawamoto, “ICA Papers Classified According to their Applications and Performances”, IEICE Trans. Fundamentals, Vol. E 86 -A, No. 3, March 2003, pp. 620 -633. • In 2002, 800 different papers have been published, these are downloadable at http: //ali. mansour. free/REF. htm IEEE UKRI Talk, De Montfort Univ. , April 2, 04 11

BSS With Matrix Constraints With a symmetric mixing matrix [C&G, 2003]: - IEEE UKRI Talk, De Montfort Univ. , April 2, 04 12

BSS With Matrix Consts. Cont. Stable Frobenius norm of the separating matrix Theorem [C&G 2003]: The learning rule where β > 0 is a scaling factor and γ(t) = trace(WT(t)F(y(t))W(t)) > 0, stabilizes the Frobenius norm of W(t) such that IEEE UKRI Talk, De Montfort Univ. , April 2, 04 13

BSS With Matrix Consts. Cont. Consequence: The modified NG descent learning algorithm, with a forgetting factor, described as with γ(t) = -trace(WT(t)[ J(W)/ W]WT(t)W(t)) > 0 has a W(t) with bounded Frobenius norm throughout the learning process. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 14

BSS With Matrix Consts. Cont. Prof. Amari’s “Leaky” NG Algorithm becomes where 0 << (1 -βγ(t)η(t)) < 1 is the leakage factor IEEE UKRI Talk, De Montfort Univ. , April 2, 04 15

BSS With Matrix Consts. Cont. Introducing a semi-orthogonality constraint so that it is possible to extract an arbitrary group of sources, say e, 1 e N. Assuming pre-whitened data and the mixing matrix A = QH, the demixing matrix We should satisfy We. A = [Ie, 0 N-e] IEEE UKRI Talk, De Montfort Univ. , April 2, 04 16

BSS With Matrix Consts. Cont. A natural gradient algorithm to find We becomes: - With initial conditions which satisfy IEEE UKRI Talk, De Montfort Univ. , April 2, 04 17

Real Convolutive Mixing Env. – Cocktail Party Problem IEEE UKRI Talk, De Montfort Univ. , April 2, 04 18

Convolutive BSS – Model Convolution Compact form: Expansion form: IEEE UKRI Talk, De Montfort Univ. , April 2, 04 19

Taxonomy of Existing Sols. To Convolutive BSS • Performing blind separation in the time domain by extending the existing instantaneous methods to conv. case • Transforming the convolutive BSS problem into multiple instantaneous (complex) problems in the frequency domain • Decomposing the system into smaller problems using, for example, a subband approach • Hybrid frequency and time domain approaches IEEE UKRI Talk, De Montfort Univ. , April 2, 04 20

Transform Convolutive BSS into the Frequency Domain DFT Convolutive BSS problem Multiple complex-valued instantaneous BSS problems IEEE UKRI Talk, De Montfort Univ. , April 2, 04 21

Mathematical Formulation In the frequency domain: - IEEE UKRI Talk, De Montfort Univ. , April 2, 04 22

De-mixing Operation IEEE UKRI Talk, De Montfort Univ. , April 2, 04 23

Constrained Optimisation and Joint Diagonalisation IEEE UKRI Talk, De Montfort Univ. , April 2, 04 24

Joint Diagonalisation Criterion Exploiting the non-stationarity of speech signals measured by the cross-spectrum of the output signals, IEEE UKRI Talk, De Montfort Univ. , April 2, 04 25

Exterior Penalty Function Approach IEEE UKRI Talk, De Montfort Univ. , April 2, 04 26

Exterior Penalty Function Approach Typical exterior penalty functions, and the shadow area represents the feasible set. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 27

Proposed General Cost Function With a factor vector κ to incorporate exterior penalty functions, our cost function becomes: - IEEE UKRI Talk, De Montfort Univ. , April 2, 04 28

Numerical Experiments • Use an exterior penalty function • Employ a variant of gradient adaptation • Utilize the filter length constraint to address the permutation problem (Parra & Spence) • System with two inputs and two outputs (TITO!) • H(z) = [{1 1. 9 -0. 75}, z-5{0. 5 0. 3 0. 2}; z-5{-0. 7 -0. 3 -0. 2}, {0. 8 -0. 1}]; D = 7, T = 1024, K = 5. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 29

Convergence Performance of the New Criterion as a function of κ IEEE UKRI Talk, De Montfort Univ. , April 2, 04 30

Room Environment Experiment • Use roommix function due to Westner • Room 10 x 10 m 3 cube • Wall reflections calculated up to fifth order, atten. factor 0. 5 • SIR is plotted as a function of length of the separating system IEEE UKRI Talk, De Montfort Univ. , April 2, 04 31

Room Environment IEEE UKRI Talk, De Montfort Univ. , April 2, 04 32

Room Environment SIR IEEE UKRI Talk, De Montfort Univ. , April 2, 04 33

Permutation Problem in FD-CBSS S 1× 0. 5 x 1 S 2× 0. 6 S 1× 0. 4 S 2 × 0. 3 x 2 FDICA IEEE UKRI Talk, De Montfort Univ. , April 2, 04 S 1 × 1. 2 34

Summary of Existing Solutions to Permut. Problem in FD-CBSS • Constraints on the filter models in the frequency domain • Using special structure contained in signals • Merging beamforming view to align solutions • Exploiting the continuity of the spectra of the recovered signals – could coupled hidden Markov Models be used? • What happens when the sources move, enter/re-enter the environment? What is the way forward? IEEE UKRI Talk, De Montfort Univ. , April 2, 04 35

Exploiting Source (Pseudo) Periodicity • W. Wang, M. G. Jafari, S. Sanei, and J. A. Chambers, “Blind source separation of convolutive mixtures of cyclostationarity”, to appear in the Special Issue on BSS, International Journal of Adaptive Control and Signal Processing, Guest Editor: Mike Davies, Queen Mary’s College, University of London • H. Swada, R. Mukai, S. Araki, and S. Makino, “A robust and precise method for solving the permutation problem of frequency-domain blind source separation”, ICA 2003, Nara, Japan, 2003, pp. 505 -510. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 36

A natural gradient update exploiting cyclostationarity • The Cyclostationary NGA uses the update equation where and p is the cycle frequency of the p-th source IEEE UKRI Talk, De Montfort Univ. , April 2, 04 37

A natural gradient update exploiting periodicity • The Periodic NGA type update equation where IEEE UKRI Talk, De Montfort Univ. , April 2, 04 38

Emerging Applications Biomedical: ECG, EEG, MEG and their integration Microarray time courses Star Trek: The Tri-corder Measurements from the nano-lab http: //www. nmrc. ie/research/transducers-group/trends. html http: //www. nanospace. systems. org/ns_2000/NS 00_Sessions. htm http: //nanomed. ncl. ac. uk/m 2 l. htm IEEE UKRI Talk, De Montfort Univ. , April 2, 04 39

Emerging Applications • T. Bowles, J. Chambers, and A. Jakobsson, “Advanced spectral estimation for the identification of cell-cycle regulated genes”, IEEE EMBS UK and RI Postgraduate Conf in Biomedical Engineering and Medical Physics, 2003. • X. Liao, and L. Carin, “Constrained independent component analysis of DNA microarray signals”, IEEE Workshop on Genomic Signal Processing and Statistics, 2002. • S-I, Lee, and S. Batzoglou, “Discovering biological processes from microarray data using independent component analysis”, Dept EE/CS, Stanford Univ. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 40

Summary • The exploitation of constrained optimisation has been fundamental to the development and application of adaptive signal processing; this process is, however, very much in its infancy in blind source separation (BSS). • Utilisation of certain a priori knowledge on the mixing matrices and the properties of the sources is likely to yield solutions to real-life SP problems. • As such, the challenge for DSP engineers in the 21 st Century, is to advance the application of BSS methods in line with methods from adaptive signal processing. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 41

Other References • A. Cichocki, and P. Georgiev, “Blind source separation algorithms with matrix constraints”, IEICE Trans. Fundamentals, Vol. E 86 -A(3), March 2003, pp. 522 -531. • J. G. Mc. Whirter, “Mathematics and signal processing”, Mathematics Today, April 2003, pp 47 -54. • W. Wang, S. Sanei, and J. Chambers, “Penalty function based joint diagonalization approach for convolutive blind source separation”, submitted to IEEE T-SP, Sept 2003. IEEE UKRI Talk, De Montfort Univ. , April 2, 04 42

Close ? ? ? Mark Twain “A man who swings a cat by its tail learns things he can learn no other way” IEEE UKRI Talk, De Montfort Univ. , April 2, 04 43