Label Tree in LargeScale Mun Jonghwan From small
Label Tree in Large-Scale Mun Jonghwan
From small to large scale 2
Learning model on Large-scale One-vs-all model … Tree structured model … 3
Label Tree • Each node has {dog, wolf, cat, tiger} {dog, wolf} {dog} {wolf} {cat, tiger} {cat} {tiger} • Label set • Classifier of children(or edge) • Each child label set is subset of its parent • Two consideration • How to split the label set(construction) • How to learn classifier(optimization) 4
Talk about… • Label Embedding Trees for Large Multi-Class Tasks(NIPS 2010) • Samy Bengio, Jason Weston, David Grangier • Fast and Balanced: Efficient Label Tree Learning for Large Scale Object Recognition(NIPS 2011) • Jia Deng, Sanjeev Satheesh, Alexander C. Berg, Li Fei-Fei 5
Label Embedding Trees for Large Multi-Class Tasks NIPS 2010 Samy Bengio et al.
Flow of constructing label tree Learning tree structure {pencil, pen, cat, tiger} Label set Cat Tiger Pencil pen Learning classifier {pen, pencil} {pencil} {cat, tiger} {cat} {tiger} 7
Tree loss • 8
Learning label tree structure • Confusion matrix {pencil, pen, cat, tiger} Cat Tiger Pencil Cat 1 0. 6 0. 12 Tiger 0. 6 1 0. 2 0. 16 Pen 0. 1 0. 2 1 0. 9 pencil 0. 12 0. 16 0. 9 1 Recursively Spectral clustering … {pen, pencil} {pencil} {cat, tiger} {cat} {tiger} 9
Learning with fixed label tree exmaple node • Relaxation 1: Independent convex problems • Relaxation 2: Tree loss optimization(Joint convex problem) 10
Label Embedding W Embedding V • 11
Label Embedding • Goal • Non-Convex joint optimization 12
Label Embedding - Sequence of convex problem Learning V Embedding ⨯ V ⨯ ⨯ · ·· Learning W 13
Experiment - result 14
Fast and Balanced: Efficient Label Tree Learning for Large Scale Object Recognition NIPS 2011 Jia deng et al.
Limitation of first paper • Learning one-vs-all classifier is costly for large-scale • Disjoint partition of classes does not allow overlap • Tree structure may be unbalanced Goal • jointly learns the splits and classifier weights • Allowing overlapping of class labels among children • Explicitly modeling the accuracy and efficiency trade-off 16
Construction of label tree {cat, fox, tiger, book, pen, note} {cat, fox, tiger} {pen, note} {book, tiger} cat fox tiger book pen note Child 1 1 0 0 0 Child 2 0 0 1 1 Child 3 0 0 1 1 0 0 • Reculsively split the node and learning weight 17
Optimization • 18
OP 1 - Optimizing efficiency with accurary constraint • Children does not need to cover all the classes in parent • Allow overlap of label sets between children 19
OP 2 - Optimizing over w given P • 20
OP 3 - Optimizing over P • 21
Summary of algorithm • Algorithm 2 is applied recursively from the root 22
Experiment 23
Experiment 24
Conclusion Samy Bengio et al. Jia Deng et al. • Fast multi-classification by • Jointly learns the splits and classifier weights optimizing the overall tree loss • Allowing overlapping splits • Embedding with the label • Explicitly modeling the accuracy and efficiency trade-off 25
Thank you 26
- Slides: 26