Costsensitive Deep Metric Learning for FineGrained Image Classification

Cost-sensitive Deep Metric Learning for Fine-Grained Image Classification Junjie Zhao and Yuxin Peng* Institute of Computer Science and Technology Peking University, China {zhaojunjie, pengyuxin}@pku. edu. cn

Guide Line • Motivation • Method • Experiment • Conclusion • Related Work Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Motivation • Fine-grained image classification: Recognize hundreds of subcategories belonging to the same basic-level category • Challenges: ① Large variances in the same subcategory Black Footed Albatross Smart fortwo Convertible ② Small variances among different subcategories Marsh Wren Rock Wren Winter Wren BMW 1 Hyundai Toyota Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Motivation • Existing methods generally adopt a two-stage learning pipeline: Localize the discriminative regions of objects Encode the discriminative features for training classifiers Zhang et al. , Part-based R-CNNs for Fine-grained Category Detection, ECCV 2014. Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Motivation • Limitation: Current methods treat all subcategories by equal cost The confusion degrees among different subcategories are extremely different Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Motivation • Contributions of cost-sensitive deep metric learning (CDML) : Confusion driven deep metric learning. Enhance the distinguish ability of the model for these confusing subcategories Weighted softmax. Learn the discriminative features for easily mis-classification subcategories and enhance representation ability of the learned model to these hard subcategories Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Guide Line • Motivation • Method • Experiment • Conclusion • Related Work Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Our cost-sensitive deep metric learning Method • Confusion Driven Deep Metric Learning • Weighted Softmax Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Confusion Driven Deep Metric Learning • Confusion Analysis Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

• Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Confusion Driven Deep Metric Learning • Number of triplet tuples for subcategory Li and Lj Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Weighted Softmax • The proposed weighted softmax loss can enhance the representation ability to the easily mis-classification subcategories via putting more costs to these hard subcategories than others – f(Ii, Li) donates the output value of the last fully-connected layer for image Ii and subcategory Li – Wi donates the probability that the images belonging to subcategories Li are mis-classified to other subcategories Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Guide Line • Motivation • Method • Experiment • Conclusion • Related Work Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

• Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Accuracy Methods Classification Accuracy (%) CDML 84. 88 STN(Jaderberg et al. NIPS 2015) 84. 10 Bilinear-CNN(Lin et al. ICCV 2015) 84. 10 Coarse-to-Fine(Yao et al. TIP 2016) 82. 90 NAC(Simon et al. ICCV 2015) 81. 01 PIR(Zhang et al. TIP 2016) 79. 34 TL Atten (Xiao et al. CVPR 2015) 77. 90 MIL(Xu et al. TIP 2017) 77. 40 Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Comparison with baselines • Confusion driven deep metric learning vs baseline • Weighted softmax vs baseline • CDML vs confusion driven deep metric learning and weighted softmax Methods Classification Accuracy (%) CDML Confusion Driven Deep Metric Learning Weighted Softmax 84. 88 Baseline 82. 33 83. 74 83. 53 Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Classification results of 20 subcategories on CUB-200 -2011 dataset • Y-coordinate donates the mis-classification rate, while the shorter bar is the better Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Guide Line • Motivation • Method • Experiment • Conclusion • Related Work Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Conclusion • The confusion driven deep metric learning approach improves the ability to model the difference among the similar and confusing subcategories emphatically. • The weighted softmax enhances the representation ability of the model for the easily mis-classification subcategories • The above two components are jointly optimized to regularize and boost each other, and it achieves the improvement for fine-grained image classification. Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Guide Line • Motivation • Method • Experiment • Conclusion • Related Work Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

• To address the problem of fine-grained image classification, object-part attention model is proposed, which is the first work to classify fine-grained images without using object or parts annotations in both training and testing phase, but still achieves promising results. • Yuxin Peng, Xiangteng He, and Junjie Zhao, “Object-Part Attention Model for Fine-grained Image Classification”, IEEE TIP 2018 • Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang, “The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification”, CVPR 2015 Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

• Considering the complementarity of text, a two-stream model is proposed to combine vision and language for learning multi-granularity, multi-view and multilevel representations Xiangteng He and Yuxin Peng, “Fine-grained Image Classification via Combining Vision and Language”, CVPR 2017 Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

• To accelerate classification speed, saliency-guided fine-grained discriminative localization is proposed, which jointly facilitates fine-grained image classification and discriminative localization Xiangteng He, Yuxin Peng and Junjie Zhao, “Fine-grained Discriminative Localization via Saliency-guided Faster R-CNN”, ACM MM 2017 https: //github. com/PKU-ICST-MIPL/Saliency-guided-Faster-RCNN_ACMMM 2017 Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

• Box constraint defines the relationship between the object and its parts, and aims to ensure that the selected parts are definitely located in the object region, and have the largest overlap with the object region • Parts constraint defines the relationship of the object’s parts, is to reduce the parts’ overlap with each other to avoid the information redundancy and ensure the selected parts are the most distinguishing parts from other categories • Xiangteng He and Yuxin Peng, "Weakly Supervised Learning of Part Selection Model with Spatial Constraints for Fine-grained Image Classification", AAAI 2017 Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Our related works • Yuxin Peng, Xiangteng He, and Junjie Zhao, “Object-Part Attention Model for Fine -grained Image Classification”, IEEE TIP 2018 • Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang, “The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification”, CVPR 2015 • Xiangteng He, Yuxin Peng and Junjie Zhao, “Fine-grained Discriminative Localization via Saliency-guided Faster R-CNN”, ACM MM 2017 【 source】 https: //github. com/PKU-ICST-MIPL/Saliency-guided-Faster-RCNN_ACMMM 2017 • Xiangteng He and Yuxin Peng, “Fine-grained Image Classification via Combining Vision and Language”, CVPR 2017 • Xiangteng He and Yuxin Peng, “Weakly Supervised Learning of Part Selection Model with Spatial Constraints for Fine-grained Image Classification”, AAAI 2017 • Xiangteng He and Yuxin Peng, "Visual-textual Attention Driven Fine-grained Representation Learning", arxiv 2017 • Xiangteng He, Yuxin Peng, and Junjie Zhao, "Fast Fine-grained Image Classification via Weakly Supervised Discriminative Localization", ar. Xiv 2017 Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM

Thank you for your attention! Homepage of our lab Source codes in Github Weakly Supervised Learning Deep of Part. Metric Selection Model with Spatial Constraints for Classification, Fine-grained Image Classification, He 2018 et al. , AAAI 2017 Cost-sensitive Learning for Fine-Grained Image Zhao et al. , MMM