Dimitrios Dimitriadis ICASSP 2019 2 3 1 J

Dimitrios Dimitriadis ICASSP 2019

2

3 [1] J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe, “Deep clustering: Discriminative embeddings for segmentation and separation, ” in ICASSP 2016. [2] Z. Chen, Y. Luo, and N. Mesgarani, “Deep attractor network for single-microphone speaker separation, ” in ICASSP 2017. [3] D. Yu, M. Kolbæk, Z. -H. Tan, and J. Jensen, “Permutation invariant training of deep models for speaker-independent multitalker speech separation, ” in ICASSP 2017. [4] M. Delcroix, K. Zmolikova, K. Kinoshita, A. Ogawa, and T. Nakatani, “Single channel target speaker extraction and recognition with speaker beam, ” in ICASSP 2018. [5] A. Ephrat et al, “Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation, ” ar. Xiv preprint ar. Xiv: 1804. 03619, 2018. [6] H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. Mc. Dermott, and A. Torralba, “The sound of pixels, ” ar. Xiv preprint ar. Xiv: 1804. 03160, 2018. [7] Z. Chen, X. Xiao, T. Yoshioka, J. Li, H. Erdogan, and Y. Gong, “Multi-channel multi-speaker overlapped speech recognition with location guided speech extraction network, ” in SLT 2018. [8] Q. Wang, et al. "Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking. " ar. Xiv preprint ar. Xiv: 1810. 04826 (2018).

4

5 Baseline system Single embedding vector Proposed system Target speaker’s mask Extractor Concatenation Mean pooling Contextdependent embedding vectors Concatenation Attention • The baseline system uses a single speaker embedding vector to inform the network the characteristics of the target speaker. Contextdependent embedding vectors Attention Embedder Target speaker’s profile Mixed speech Embedder Non-target speakers’ profiles

6 Baseline system Single embedding vector Proposed system Target speaker’s mask Extractor Concatenation Mean pooling Contextdependent embedding vectors Concatenation Attention • The baseline system uses a single speaker embedding vector to inform the network the characteristics of the target speaker. Contextdependent embedding vectors Attention Embedder Target speaker’s profile Mixed speech Embedder Non-target speakers’ profiles • In the proposed system, mean pooling is replaced by attention to estimate the context dependent bias signals.

7 Baseline system Single embedding vector Proposed system Target speaker’s mask Extractor Concatenation Mean pooling Contextdependent embedding vectors Concatenation Attention • The baseline system uses a single speaker embedding vector to inform the network the characteristics of the target speaker. Contextdependent embedding vectors Attention Embedder Target speaker’s profile Mixed speech Embedder Non-target speakers’ profiles • In the proposed system, mean pooling is replaced by attention to estimate the context dependent bias signals. • Competing speakers’ information are fed into the network as a contrast.

8 A bias vector is a weighted sum of profile embedding vectors go to extraction sub-network Target speaker’s profile embedding From embedding sub-network Weights are obtained from similarity Weights are time-dependent and usually sparse From embedding sub-network

9 go to extraction sub-network Computing speakers’ embedding Target speaker embedding Mixed log magnitude

1 0

1 1

1 2 Systems Target speaker profile (# sentences) Competing speaker profile (# sentences) Other speakers used as competing speakers (# other speakers / # sentences per speaker) SDR (d. B) Baseline 1 0 0 9. 4 5 0 0 9. 8 1 1 0 11. 5 1 1 1/1 11. 1 1 1 2/2 10. 9 2 2 0 11. 9 3 3 0 12. 0 5 5 0 12. 1 5 5 1/1 12. 1 5 5 2/2 12. 0 Proposed Effect of using attention and contrast bias

1 3 Systems Baseline Proposed Target speaker profile (# sentences) Competing speaker profile (# sentences) Other speakers used as competing speakers (# other speakers / # sentences per speaker) SDR (d. B) 1 0 0 9. 4 5 0 0 9. 8 1 1 0 11. 5 1 1 1/1 11. 1 1 1 2/2 10. 9 2 2 0 11. 9 3 3 0 12. 0 5 5 0 12. 1 5 5 1/1 12. 1 5 5 2/2 12. 0 • Moderate performance degradation due to uninvolved contrast bias, but still better than baseline. • Advantage: we do not know exactly who is the competing speaker, as far as the competing speaker is included in the contrast bias set.

1 4 Systems Target speaker profile (# sentences) Competing speaker profile (# sentences) Other speakers used as competing speakers (# other speakers / # sentences per speaker) SDR (d. B) Baseline 1 0 0 9. 4 5 0 0 9. 8 1 1 0 11. 5 1 1 1/1 11. 1 1 1 2/2 10. 9 2 2 0 11. 9 3 3 0 12. 0 5 5 0 12. 1 5 5 1/1 12. 1 5 5 2/2 12. 0 Proposed • Improvement due to longer profile person. • Expected to get better performance if we get more data about the speakers.

1 5 Weights are usually sparse Only a few inventory frames are selected for each mixed frame. Target Competing speaker Get attention weights Mixed Bystander speaker Mix Get attention weights Competing Speaker selectivity via attention Only the competing speaker gets a lot of attention weights, while the other speaker that doesn’t contribute to the mixed speech has close-to-zero weights.

1 6

1 7

1 9 Matrix Multiplication Target weight matrix as in Eq. (3) Target bias Target Competing speaker Content selectivity via attention each frame gets context-dependent biases which tell the extractor what the target speaker’s voice look like for the phonemes in the mixed speech. Embedding & Eq. (2 -3) Mixed Other speaker Mix Embedding & Eq. (2 -3) Competing weight matrix as in Eq. (3) Speaker selectivity via attention Only the competing speaker gets a lot of attention weights, while the other speaker that doesn’t contribute to the mixed speech has almost empty weights. Competing bias Matrix Multiplication