Leicester Data Mining Team Lt R Ivan Tyukin

Leicester Data Mining Team. Lt. R: Ivan Tyukin, Jeremy Levesley, Alexander Gorban, Evgeny Mirkes 2

Two main streams of AI Rule-based AI (Chess Champion) Machine Learning AI (Go Champion) Hybrid AI systems 3

Plan • BIG Legacy AI Systems and their mistakes • Augmenting artificial intellect: a conceptual framework • Self-esteem and error functions for neural networks; • Ensemble learning and distribution of problems between local experts; • Adaptive coding and self-supervising learning; • Organisation of social networks of neural agents and non-iterative knowledge transfer. 5

GOOGLE, Amazon продают • Услуги нейронных сетей по распознаванию; • Услуги стартапам по выращиванию нейронных сетей: «Google Cloud's AI provides modern machine learning services, with pre-trained models and a service to generate your own tailored models. » • Услуги, использующие нейронные сети: «Major Google applications use Cloud machine learning, including Photos (image search), the Google app (voice search), Translate, and Inbox (Smart Reply). » 10

The World of BIG Legacy AI Systems Free (open-source) AI systems that are already created and form crucial parts of existing solutions/service • Face. Net at Git. Hub (https: //github. com/davidsandberg/facenet) • VGG-Face (http: //www. robots. ox. ac. uk/~vgg/software/vgg_face/) • Google Inception 3 (https: //github. com/tensorflow/models/tree/master/inception) • etc They are built using massive volumes of data, and have large number of “internal” variables

The World of BIG Legacy AI Systems is spreading to small low-power devices too https: //youtu. be/w 6 bm 2 V 7 LSCs Inception v 3 running on Raspberry Pi 3 platform (Lukas Biewald, September 21, 2016)

The World of BIG Legacy AI Systems AI systems that are already created and form crucial parts of existing solutions/service Commercial • Google (Google Cars, Face. Net, Deep. Mind) • Facebook (Deep. Text, face recognition API) • Amazon (Recognition, Alexa) • Apple (Siri, face recognition phone unlock) • IBM (Watson, face recognition API) • NEC (face recognition) They are built using massive volumes of data, and have large number of “internal” variables

All AI Systems Make mistakes! 27

AUGMENTING ARTIFICIAL INTELLECT: A CONCEPTUAL FRAMEWORK A paraphrase of Douglas C. Engelbart DC. Augmenting human intellect: a conceptual framework (1962). PACKER, Randall and JORDAN, Ken. Multimedia. From Wagner to Virtual Reality. New York: WW Norton & Company. 2001: 64 -90. Conceptual framework of the Internet 28

Two papers published simultaneously in 1991 • SE Gilev, AN Gorban, EM Mirkes, Small experts and internal conflicts in learning neural networks. Akademiia Nauk SSSR, Doklady 320 (1), 220 -223, 1991. (GGM 1991) • RA Jacobs, MI Jordan, SJ Nowlan, GE Hinton, Adaptive mixtures of local experts, Neural computation 3 (1), 79 -87, 1991. 29

Interpretation of outputs (classifier) inputs out 1 out 2 Winner takes all 1 out 2 Classifiers are used as a simple class of AI systems Class 1 out 1 130

GGM 1991 Interpretation and error function out 2 1 out 2 Class 2 error 1 Class 2 error Class 1 error Class 2 Class 1 error Class 1 1 out 2 Class 1 Perfect training out 1 1 Class 1 ε-th of r e leve l elia The b i e lity by rror 1 out 2 out 1 1 f t Class 2 (an he out unctio d b y th put in n is de e le terp t vel reta ermin e of r Class 1 elia tion d out 1 bili ty) 1 Perfect training 31

GGM 1991 Benefits out 2 1 1. Simplification of learning tasks (less neurons and less time is needed for training); 2. New useful characteristic appears: the level of self-esteem s (the distance from the diagonal): If s/ε>1 then the system “is sure” in the answer (the training is assumed to be validated with given ε) Class 2 error Class 1 error Class 2 Class 1 ε out 2 out 1 1 1 Correct, Incorrect, for sure with Incorrect, with doubt without doubt ε ε Validation histogram Class 2, I am sure s Class 2 s/ ε >1 s/ε <1 Class 1, but I am not so sure Class 1 ε out 1 1 32

GGM 1991 Idea 1 Let many agents share the problem fields. We teach agents: • To solve their own problems correctly with high level of self-esteem ε; • To solve problems owned by other agents correctly or “incorrectly but with lower level of self-esteem δ<ε”. 33

GGM 1991 Supervising teaching • Take a task • Select the owner (an agent with the best correct solution, or with the smallest mistake) • Teach the owner by a step or several steps of gradient descent of the error function with selfesteem ε; • Teach non-owners with a step or several steps of gradient descent of the error function with selfesteem -δ (with δ<ε). • Take the next task 34

GGM 1991 1 D plots of MSE (for teaching) Estimate e Own te mat esti tima ner r es -ow Non e Incorrect class -δ Diagonal Ideal solutions for owners ε Correct class Ideal solutions for non-owners 35

GGM 1991 A piece-wise linear estimate Estimate e Own te mat esti tima ner r es -ow Non e Incorrect class -δ Diagonal Ideal solutions for owners ε Correct class Ideal solutions for non-owners 36

GGM 1991 Functioning: the agent with maximal self-esteem gives the final answer The most self-confident answer: Winner takes all Class 2 The answer we accepted Diagonal Class 1 The range of answers 37

GGM 1991 Floating classes, unsupervised dynamic coding and skills updating In real life, the classes “float” in time and the classification model should also float. • Take a new sample; • Label of this new example by the “winner takes all” rule; • Use this label for a step of supervising teaching (for the owner and for non-owners). 38

GGM 1991 Collective ownership • In supervising learning the owners of a sample are k agents with k best correct solution, or with the smallest mistake. Owners: k best answers Incorrect class Diagonal Correct class The range of answers 39

GGM 1991 Estimates for collective ownership Estimate 1 st owner estimate Non 2 nd owner estimate ner -ow ……………. e mat esti kth owner estimate Incorrect class It remains SOMs in the output space -δ Diagonal Ideal solutions for owners ε Correct class Ideal solutions for non-owners 40

GGM 1991 Functioning with collective ownership: the weighted voting of owners The owners: k most self -confident answers Class 2 The answer we accepted after voting. The weight of the vote depends on the relative selfconfidence of the agent. Diagonal Class 1 The range of answers 41

GGM 1991 Floating classes, unsupervised dynamic coding and skills updating for collective ownership • Take a new sample; • Find k most self-confident agents for this sample • Label this new example by the voting of owners; • Use this label for a step of supervising teaching (for the owners and for non-owners). TOO MUCH ITERATIVE LEARNING 42

43

The problem of corrections of legacy systems Inputs Internal signals Outputs A legacy system It works well but sometimes makes mistakes. Corrections are needed!

The problem of corrections of legacy systems Inputs Internal signals Outputs Correction Corrector should take some input, internal, and output signals of the system and produce correction

One-shot AI Knowledge Transfer At the unsupervised stage the agents are considered as legacy AI systems Corrector building blocks: linear functionals or small neural networks 46

Corrector should: • be simple; • not change the skills of the legacy system; • allow fast non-iterative learning; • allow correction of new mistakes without destroying of previous corrections. Thus, it has to separate mistakes from correctly solved examples and correct mistakes!

Geometrical preliminaries

Concentration of the volume Almost all volume appears at r≈1 4 n=3 2 n=64 n=128 n= n= 16 2 n= 8 Dim n= o si en 1 = n n Volume of high-dimensional ball is concentrated near its border (sphere) Radius r

Waist concentration In high dimension, the volume of the ball is concentrated near its equator (each equator) Corollary: Two random unit vectors are almost orthogonal almost for sure (the angle between them is close to π/2 with high probability).

Between Gibbs and Levy • • Levy systematically used asymptotic equivalence of equidistribution on sphere and Gaussian distribution in high dimensions. This is a particular case of equivalence of microcanonical and canonical ensembles (Gibbs, 20 years before Levy). Microcanonical ensemble is a statistical ensemble where the total energy of the system and the number of particles in the system are fixed. It is equidistribution on the isoenergetic surface in Liouville measure. Canonical ensemble is given by Boltzmann’s distribution and maximizes entropy, the temperature is known but the energy is not fixed. These ensembles are equivalent in thermodynamic limit (high dimension) – demonstrated by Gibbs!

Ensemble equivalence n=2 =4 =8 =16 =32 =64 =128 Normalised to unit mean and max

Ensemble equivalence n=2 =4 =8 =16 =32 =64 =128 Probability density has maximum at the origin but the distribution of radius is concentrated far from the origin. Normalised to unit mean and max

Stochastic separation theorems in high dimensions Expected ε IS NOT small in high dimension? ε Extreme points In high dimension, with high probability in an exponentially large random set all points are extreme ones? ? ? !

55

AI Knowledge Transfer from teacher to a student A teacher Legacy AI system processes the input signals related to the input seen by the student and returns expert outputs “Teacher” Legacy AI system “Student” Feedback 56

Knowledge transfer between AIs In our experiments: • The teacher AI, AIt, was modelled by a deep Convolutional Network, Res. Net 18 with circa 11 M trainable parameters. • The teacher network was trained on a “teacher" dataset comprised of 5. 2 M non-pedestrian (negatives), and 600 K pedestrian (positives) images. • The student AI, AIs, was modelled by a linear classier with 2016 trainable parameters. • The values of these parameters were the result of AIs training on • a student" dataset, a sub-sample of the “teacher" dataset comprising of 130 K negatives and 55 K positives. • This choice of AIs and AIt systems enabled us to emulate interaction between edge-based AIs and their more powerful counterparts that could be deployed on larger servers or computational clouds.

Knowledge transfer between AIs: False Positives induced by the teacher AI These errors contain genuine false positives (images 12, 23 -27) as well as mismatches by size (e. g. 1 -7), and lookalikes (images 8, 11, 13, 15 -17).

Knowledge transfer between Ais: Student’s errors before and after teaching • Red circles show true positives as a function of false positives for the original student linear classier. • Blue stars and Green triangles correspond to AIs after correction of false positives by the teacher (two different algorithms or errors clustering). • Black squares correspond to AIs after correction of false positive followed by correction of false negative.

Non-iterative knowledge transfer from the owners conference to all agents, who find the sample interesting • Take a new sample; • Find q (q>k) most self-confident agents for this sample (the agents who find this sample “interesting”; k most self-confident of them are owners) • Label this new example by the voting of owners; • Create a non-iterative corrector for the agents who are interested in this sample but make a mistake. • If a “human supervisor” makes labelling then it can be used for corrections of all the agents who are interested in this sample but make a mistake. 60

Interiorisation of knowledge • After collection of correctors, the interiorisation of the collected knowledge can be used. • Interiorisation of knowledge is its incorporation into agent’s inner structure. • Interiorisation can be organised as an iterative supervised learning procedure that uses the agent with correctors for sample labelling as the supervisor. • Interiorisation is performed for all agents separately and independently, in parallel. • After the individual interiorisation procedure, the agents return to the network learning. 61

Correctors Agents ` ` ` Network learning Retu ` ` rn to en d u t S ` t the netw ork Knowledge interiorisation Supervisor ` 62

Results and problems +AI with self-esteem are efficient even for standard learning tasks; +The multiagent systems with self-esteem can solve the problems of growing complexity; +These systems combine the supervising pre-learning and self-learning of the networks with corrections and knowledge transfer; +Interiorisation of knowledge is its incorporation into agent’s inner structure for further learning and work. ? The scalability of the method to big and very big heterogeneous problem spaces has to be analysed. 63