Graph Based MultiModality Learning Hanghang Tong Jingrui He

Graph Based Multi-Modality Learning Hanghang Tong; Jingrui He Carnegie Mellon University Mingjing Li Microsoft Research Asia ACM/Multimedia 2005

Outline o o Motivation Graph-based Semi-supervised learning n n o o o Methods o The Linear Fusion Scheme o The Sequential Fusion Scheme Justifications o Similarity Propagation o Bayesian Interpretation Graph-based un-supervised learning Experimental Results Conclusion 2021/12/14 ACM/Multimedia 2005 2

Motivation o Multi-Modality in Multimedia n n n o Video: Digital Image: color vs. Web Image: content vs. surrounding text Traditional methods n n n All Vector Model based ! Linear combination; super-kernel… Co-Training… Multi-view version of EM, DBSCAN… 2021/12/14 ACM/Multimedia 2005 3

Motivation (conts) o Two Key issues n n o A more recent hot topic n o How to learning within each modality How to combine… Graph-based learning o Spectral Cluster; Eigen Map o Manifold Ranking… Explore graph-based method in the context of multi-modality! 2021/12/14 ACM/Multimedia 2005 4

Notation o o n data points c classes, Two modalities: a and b One Affinity Matrix for each modality: n n o o for modality a (nxn) for modality b (nxn) Labeling Matrix: Vectorial Function: Learning task: s 2021/12/14 ACM/Multimedia 2005 (nxc) 5

Basic Idea o What is a ‘good’ vectorial function F? n As consistent as possible with o o Info from modality a Info from modality b Info from Labeled points Y How to? n Take into account the various constraints by optimization 2021/12/14 ACM/Multimedia 2005 6

Linear Fusion Scheme Constrains. from modality b Constrains. from o modality a Optimization strategy o Optimization Solution n Iterative form n Closed form 2021/12/14 Constrains. from Labels Y Converge ACM/Multimedia 2005 7

Sequential Fusion Scheme o o Optimization strategy Optimization Solution n Constrains. from modality a and Y Constrains. from modality b and Iterative form Converge n Closed form 2021/12/14 ACM/Multimedia 2005 8

Similarity Propagation o Taylor expansion (linear fusion) Initial Label > Propagate Y by a and b; > Fuse the result by weighted sum o > Further propagate similarity by a and b; > Fuse the result by weighted sum Similar result for sequential form 2021/12/14 ACM/Multimedia 2005 9

Bayesian Interpretation o Optimal F by MAP (linear form): o Assuming: n n o Conditional pdf Prior by modality a Prior by modality b Fuse prior by The above setting leads to… 2021/12/14 ACM/Multimedia 2005 10

Extension to Un-Supervised Case o Compare n n o o Quite similar ! For one modality: For two modalities (linear form): Graph Laplacian Learning n Linear Form: n Sequential From: Independent on Y ! Feed it the spectral cluster or embedding… 2021/12/14 ACM/Multimedia 2005 11

Experimental Results: Coral Image Sequential Form Linear Form Treat as one modality Color Hist Texture 2021/12/14 ACM/Multimedia 2005 12

Experimental Results: Web Image Linear Form Sequential Form Treat as one modality Content Fea Surrounding Text 2021/12/14 ACM/Multimedia 2005 13

Conclusion o o o Study multi-modality learning by graph based method Propose two schemes for semisupervised learning Extend them to un-supervised learning 2021/12/14 ACM/Multimedia 2005 14

Q&A The End Thanks 2021/12/14 ACM/Multimedia 2005 15