Intelligent Systems Laboratory of Moscow Research Center Mazurenko

Intelligent Systems Laboratory of Moscow Research Center Mazurenko Ivan, 2019. 07. 03 Mazurenko. Ivan 1@Huawei. com Security Level:

Huawei Moscow Research Center Concept is algorithm design based on strong Mathematical knowledge supported by Leading Russian Universities and Institutes of Russian academy of Science Moscow RC 200+ Intelligent systems Lab. IT Algorithm Lab. RTT Algorithm Lab. Media Algorithm Lab. Mathematic Modeling & Optimization CC Coding Algorithm Competence center IRF Algorithm Competence Center 40+ Mathematical Researchers from Lomonosov Moscow State University, MIPT, Skoltech, HSE, Bauman and other Universities, including 25 Ph. Ds, 2 Russian Doctors of Science + 20 interns

Intelligent Systems Lab Directions • Intelligent systems lab • Video intelligence group Video surveillance • Face. ID, Dataset augmentation, AI Chip, … • Camera AI group Smartphone camera • Segmentation, Denoising, Anti-distortion, … • Big data group • Graph mining, Content aware video compression, Video surveillance, … • • 3 Image search group SSD group Quantum ECC group ADAS group – new Huawei Confidential

Video Intelligence Direction Easy to see (IPC imaging) Accurate capture (Target snapshot) Recognizable (Person/vehicle identification) Fast search (Person/vehicle search) Easy to understand (Content and behavior understanding) Well coding(Transcoding and enhancement) Easy to transfer (Conference and live broadcast) Cloud transcoding Valueable Features Scenario identification Facial snapshot License plate Face Recognition Abnormal/Normal video Dark light imaging Pedestrian snapshot Car brand & color Facial attribute Multi-focal multi-camera Imaging Vehicle snapshot local feature Human body attribute Video image stitching Pedestrian snapshot Key Technologies Calculation Human body/vehicle image search Algorithm training cluster + evaluation, segmentation, identification, retrieval, and Ø D chip Hi 35 XX joint development data-distributed training Auto. ML Ø GPUDSPIVEARM FPGAX 86 algorithms optimization Ø Transcoding and packet loss Live video on demand (Vo. D) Video network collaboration Multi-category Fight, gather. . . Data clustering Perceptual coding (PVC) Device-cloud congestion control Who do What KNN Desktop Coding FEC/ARQ What do what at where Intra-class segmentation Enhanced and quality detection ROI transmission Data Ø Detection, tracing, quality Ø Multi-machine and multi-GPU Feature quantization Ø Data design and data production + system design Ø Semi-automatic and automatic marking of data Ø Data increase and data cleaning + Identification System Ø Interactive recognition system Ø IPC imaging and snapshot system Ø Video cloud processing and transmission system Ø RGB+IR fusion and AI + market practices in Guangzhou, Hefei, and Xinjiang. Ø Cloud. BU Yingkou live broadcast practice Ø Video Conferencing Company Office Practice Ø ……. enhancement HUAWEI TECHNOLOGIES CO. , LTD. Ø Take snapshots and identify Ø Vehicle Association device-cloud collaborative concealment Market practice System design Huawei Confidential 4

On Fundamental Mathematical Problems in Deep Learning Huawei Moscow Research Center, Intelligent Systems Lab Mazurenko Ivan, Petyushko Alexander 2019. 05. 31 Security Level:

Contents • Introduction • 1 Expressiveness • 2 Convergence • 3 Generalization • 4 Robustness • Related math problems • 5 Metric learning • 6 Feature mapping 6 Huawei Confidential

1 Expressiveness Can Neural network (and Convolutional neural network in particular) represent any continuous function with any desired precision? • In 1957, Andrey Kolmogorov (USSR, Moscow State University) proved a Universal representation theorem [1] for continuous functions: Any multivariate continuous function can be expressed as a finite sum of continuous functions of a single variable • This problem closely relates to 13 th Hilbert’s problem and may be considered as a fundamental theoretical basis for the whole Neural networks theory. • In 1989 George Cybenko (Dartmouth college, USA) proved a Universal approximation theorem [2] for multi-layer feedforward neural networks called “perceptrons”: Any continuous function on a compact domain can be approximated with any given precision by a 2 -layer perceptron with monotonically increasing nonconstant bounded activation function. • Andrey Kolmogorov George Cybenko The theorem was independently proven by the authors of [3]. • Some recent works, like a paper by Dmitry Yarotsky (Skoltech, Russia) [4] and a work of Ding-Xuan Zhou (University of Hong Kong) [5] provide universal approximation theorems for special classes of convolutional neural networks • But the universal approximation problem for Convolutional neural networks is not solved yet. [1] Kolmogorov, A. N. (1957). “On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition”. Doklady Akademii Nauk SSR, 114, 953 -956. [2] Cybenko, G. (1988) Approxtmation by superpositions of a sigmoidal function (Tech. Rep. No. 856). Urbana, IL: University of Illinois Urbana-Champaign Department of Electrical and Computer Engineering. [3] Hornik, K. , Stinchcombe, M. , & White, H. (1988) “Multilayer feedforward networks are universal approximators” (Discussion Paper 88 -45). San Diego, CA: Department of Economics, university of California, San Diego. [4] Yarotsky, Dmitry. "Universal approximations of invariant maps by neural networks. " ar. Xiv preprint ar. Xiv: 1804. 10306 (2018). [5] Zhou, Ding-Xuan. "Universality of Deep Convolutional Neural Networks. " ar. Xiv preprint ar. Xiv: 1805. 10769 (2018). 7 Huawei Confidential Dmitry Yarotsky ?

Expressiveness of different DNNs • Multi-Layer Perceptron • Convolutional network • Graph neural network • CNN with discrete weights • Binary Neural Network • Capsule Neural Network • Impulse Neural Network • Memristor Neural Network • Open questions: What is expressiveness of each of these neural network classes? How to train discrete (binary) NN? How to automatically convert DNN from one Huawei Confidential class to another? What is the optimal bit width for discrete DNNs? 8

2 Convergence Do Gradient Descent (GD) and Stochastic Gradient Descent (SGD) converge to the global optimum? • In 1957 Frank Rosenblatt (USA) [1] proposed a concept of Perceptron and a heuristic perceptron training algorithm. • In 1962 Albert Novikoff (Stanford, USA) proved a Convergence theorem [2] for it: perceptron training converges after making not more than R 2/G 2 updates (where R is the norm of an input and G is the margin between linearly separable classes). • In recent works (2017 -2019) multiple researchers provide theorems that prove approximate convergence of Gradient Descent [3] and Stochastic Gradient Descent [4, 5, 6] with polynomial number of iterations for Convolutional neural networks with constraints and special classes of CNNs (e. g. Res. Net based). • Nevertheless, convergence theorem for general case of Convolutional neural networks is not proven yet. Frank Rosenblatt Albert Novikoff ? [1] Rosenblatt, Frank (1957), The Perceptron--a perceiving and recognizing automaton. Report 85 -460 -1, Cornell Aeronautical Laboratory [2] Novikoff, A. B. J. (1962). On convergence proofs on perceptrons. Symposium on the Mathematical Theory of Automata, 12, 615 -622. Polytechnic Institute of Brooklyn. [3] Du, Simon S. , et al. "Gradient descent finds global minima of deep neural networks. " ar. Xiv preprint ar. Xiv: 1811. 03804 (2018). [4] Allen-Zhu, Zeyuan, Yuanzhi Li, and Zhao Song. "A convergence theory for deep learning via over-parameterization. " ar. Xiv preprint ar. Xiv: 1811. 03962 (2018). [5] Difan Zou, Yuan Cao, Dongruo Zhou, Quanquan Gu ``Stochastic Gradient Descent Optimizes Over-parameterized Deep Re. LU Networks’’ ar. Xiv preprint ar. Xiv: 1811. 08888 (2018) 9 Huawei Confidential [6] Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang, Vahid Tarokh. “SGD Converges to Global Minimum in Deep Learning via Star-convex Path”. ar. Xiv preprint ar. Xiv: 1901. 00451 (2019)

DNN Convergence – Open Problems • How to compare two neural networks taking into account instability of training procedure (dependency of random weights initialization, training rate change policy etc. )? • When to stop the training procedure to prevent training set overfitting? • How to accelerate the training procedure? (tensor train, proper weights initialization, multiple GPU for data and model parallelization, tensor neuro chips, analog memristor-based networks, . . . ) 10 Huawei Confidential

3 How to select the optimal complexity of the model to reach its maximal Generalization generalization ability while preventing overfitting on the training data? • Vladimir Vapnik Alexey Chervonenkis Te st Tr n ai da ta ? ta da Sanjeev Arora Dmitry Yarotsky [1] Vapnik, V. N. ; Chervonenkis, A. Ya. (1971). "On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities". Theory of Probability & Its Applications. 16 (2): 264. [2] Vapnik, Vladimir (2000). The nature of statistical learning theory. Springer 11 Huawei Confidential [3] Arora, Sanjeev, et al. "Stronger generalization bounds for deep nets via a compression approach. " ar. Xiv preprint ar. Xiv: 1802. 05296(2018). [4] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Proceeding of the 28 th Conference on Learning Theory (COLT), 2015 b. [5] Behnam Neyshabur, Srinadh Bhojanapalli, David Mc. Allester, and Nathan Srebro. A pacbayesian approach to spectrally-normalized margin bounds for neural networks. ar. Xiv preprint ar. Xiv: 1707. 09564, 2017 a. Generalization error

DNN Generalization – Open Problems • How to estimate the optimal number of DNN weights? (analog of VC-dimension for neural networks) • How to automatically find the optimal DNN architecture for the given task? • Examples: Genetic architecture search, Monte-Carlo architecture search, Gradient descend-based architecture search methods • How to automatically select the best hyper parameters (parameters of loss functions, learning rate of SGD, weights initialization etc. )? • How to accelerate the inference of DNN? (quantization, pruning, tensor decomposition, . . . ) 12 Huawei Confidential

4 Robustness How can we guarantee that output distributions of CNN are close when distributions of input images are close? • Recently, Adversarial Examples [1] for Convolutional Neural Networks were discovered by Christian Szegedy (Google AI, USA). Two images are called Adversarial if they are very close (almost invisible to human eye) but provide significantly different output if sent to the input of some CNN. So called Adversarial Attacks (including Black Box Attacks and Real World Attacks) can fool the neural network by slightly changing the input images that may lead to incorrect classification of NN inputs. • The problem of Adversarial Defense is closely connected to the problem of calculating distance between two probabilistic distributions: we may guarantee robustness of the Neural Network if training it with special types of losses, based on calculating distance between two distributions. Such methods include KL-divergence / Relative entropy [2], Inception score [3], Frechet distance [4] etc. • In 2017 Martin Arjovsky (New York University, USA) et al suggested a to use Wasserstein metric named after Russian mathematician Leonid Vaseršteĭn who introduced this concept in 1969. This novel idea was used in a new Wasserstein Generative Adversarial Networks [5] which guarantee their training convergence. • New mathematical theory is needed to be developed for providing a guaranteed defense of any type of adversarial attacks. [1] Szegedy, Christian, et al. "Intriguing properties of neural networks. " ar. Xiv preprint ar. Xiv: 1312. 6199(2013). [2] Kullback, Solomon, and Richard A. Leibler. "On information and sufficiency. " The annals of mathematical statistics 22. 1 (1951): 79 -86. [3] Salimans, Tim, et al. "Improved techniques for training gans. " Advances in Neural Information Processing Systems. 2016. [4] Heusel, Martin, et al. "Gans trained by a two time-scale update rule converge to a local nash equilibrium. " Advances in Neural Information Processing Systems. 2017. 13 Huawei Confidential [5] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein gan. " ar. Xiv preprint ar. Xiv: 1701. 07875 (2017). Christian Szegedy Leonid Vaserstein Martin Arjovsky ?

DNN Robustness – Open Problems • How to train DNN in a robust way to defend from all known adversarial attacks? • How to compare similarity of distributions on the output of DNN and on its input? • How to evaluate the quality of Generative adversarial networks (GANs)? • How to learn the topological properties of manifolds for input images / DNN outputs? 14 Huawei Confidential

Face. ID Attacks: Geometrical & Real-world Ivan 15 Huawei Confidential Alex Impersonation Dodging (no recognition)

Generative Adversarial Networks (GANs) A generative adversarial network (GAN) is a class of machine learning systems. Two neural networks contest with each other in a zero-sum game framework. This technique can generate photographs that look at least superficially authentic to human observers, having many realistic characteristics. It is a form of unsupervised learning. The generative network G generates candidates while the discriminative network D evaluates them. The contest operates in terms of data distributions. Typically, the generative network learns to map from a latent space to a data distribution of interest, while the discriminative network distinguishes candidates produced by the generator from the true data distribution. The name "GAN" was introduced by Ian Goodfellow (Google AI) et al. in 2014. Their paper popularized the concept and influenced subsequent work. Ian Goodfellow 16 Huawei Confidential

Which of these two face images is real? 17 Huawei Confidential

Which of these two face images is real? 18 Answer: both are fake and generated by GANs! Huawei Confidential

Problem: How to Evaluate the GAN? • Which of these two GANs is better? 19 Huawei Confidential

Manifold hypothesis for AI l Manifold hypothesis: reason why NN work as classifiers p Almost all RN consists of invalid (from the point of view of real distribution) samples n p l Random image is almost always a noise All interesting cases are concentrated along a small number of low dimensional manifolds Example: p p p Initial image samples in multidimensional space of Height * Width (where h, w could be thousands) are hard to (linearly) separate: they are entangled NN aim: to disentangle classes using inner representation of a smaller dimension (on a manifold) If we are working on a manifold of a much smaller dimension this task could be solved much easier HUAWEI TECHNOLOGIES CO. , LTD. Huawei Confidential 20

5 Related problem - Metric Learning l We need to minimize average difference between maximal similarity of images from different classes and minimal similarity of images from the same class HUAWEI TECHNOLOGIES CO. , LTD. Huawei Confidential 21

6 Related problem - Feature Mapping l We need to find a mapping transform M which minimizes the mean square distance between similarities of the target network F 2 and similarities transformed from the source network F 1 using this mapping M HUAWEI TECHNOLOGIES CO. , LTD. Huawei Confidential 22

7 Other - Disentangled Representations l l Task is related to all embedding search / classification problems (such as Face. ID) The idea is to propose the training method and / or NN architecture so that: p There exists a robust definite mapping between some elements of NN embedding vector and semantic features of the objects we are classifying n p e. g. for Face. ID: eyes color, skin color, hair length etc Reference paper: “Towards a Definition of Disentangled Representations” We need to find a mapping between embeddings and semantic features of objects. The task is tightly connected to feature mapping and manifold learning HUAWEI TECHNOLOGIES CO. , LTD. Huawei Confidential 23

8 Other - Video face recognition • Background • Single image is widely adopted for face recognition in both academia and industry. However, much information is lost after selecting a key frame from the original sequence. Will it be helpful by recognizing the video sequence as a whole? If so, how? Key frame selection Feature extraction [0. 01, 0. 03, …, 0. 05] [0. 02, 0. 04, …, 0. 06] • Problem description • Input • Cropped face image sequence of the same person • Task • Find a single feature to represent the video sequence, and use it for 1: N face identification • Goal • Improve performance for scenarios where single image is inadequate, such as low-resolution, blur and large pose HUAWEI TECHNOLOGIES CO. , LTD. Huawei Confidential 24

Problems in Deep Learning Ex • Automatic optimal architecture search • Automatic hyper parameter search • Model compression ib ss e pr y t ili n o C • • • 25 Adversarial attacks Black box attacks Real world attacks Adversarial defense Identity preserving dataset augmentation Huawei Confidential R u b o st ne ss • Fast convergence algorithms • Training dataset cleanup • Single/few shot learning e c • Deep online learning n e • Loss functions research g r • NN fine tuning ve G en er al iz at io n • Optimal selection of network size • Optimal initialization • Overfitting prevention • Validation and testing methodology

Thank you. Bring digital to every person, home, and organization for a fully connected, intelligent world. Copyright© 2018 Huawei Technologies Co. , Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.