Nizhni Novgorod State University Faculty of Computational Mathematics

Nizhni Novgorod State University Faculty of Computational Mathematics and Cybernetics Information Technologies Laboratory Parallel Implementation of the Probabilistic Network Library (PNL) Labutina A. Software department

Project Goals • Implement a general purpose toolbox for working with probabilistic network models • Open source project placed on the Web-site Source. Forge to provide easy access for research and contributions 2

Project Team • Development: – Intel Russia Research Center • Victor Eruhimov Igor Chikalov, Anton Pegushin, Maria Lyashko, Alexander Brusin, Denis Lagno, Vasily Tarasov, Sergey Molinov – University of Nizhni Novgorod • Sergey Belov, Victor Gergel, Alexander Sysoev, Olga Abrosimova, Anna Labutina, Andrei Senin, Roman Vinogradov, Alexander Gergel, Elena Chernyshova, Alexey Bader, Sergey Sidorov • Management: Gary Bradski, Denver Dash, Intel Santa Clara, Microprocessor Research Lab • Validation and Developmment – Intel China Research Center – Intel Architecture Laboratory • Consultant: Kevin Murphy, MIT AI lab 3

PNL Functionality • Probabilistic Networks – – – Bayesian Networks (BNet) Dynamic Bayesian Networks (DBN) Markov Random Fields (MRF) Limited Memory Influence Diagrams (LIMID) Diagnostics Networks • Probabilistic Distributions – Discrete (tabular) • Dense representation • Sparse representation (hash) – Multivariate Gaussian, Conditional Multivariate Gaussian – Decision tree – Softmax (including sigmoid distribution) 4

Inference and Decision Making • Exact and Approximate Inference Engines – – – – Naïve Bayesian inference engine Junction tree inference engine Loopy believe propagation Gibbs sampling 1. 5 junction tree (DBN) BK inference Variational approximation • Decision Making – Single policy updating (LIMID) – Value of information (diagnostic networks) 5

Learning Algorithms • EM Parameter Learning – – – Can use any applicable inference engine for missing data support Generalized EM Iterative proportional fitting for MRFs Conjugated gradients Iteratively reweighed least squares • Bayesian Parameter Learning – Dirichlet (K 2) priors – Wishart priors • Structure Learning – BD metrics, BDeu priors, BIC, AIC scores – Predictive assessment 6

Languages, Interfaces and Platforms • Programming language is C/C++ • R interface • Genie interface • Matlab interface (generated automatically from C++ API using SWIG environment) • Windows and Linux versions 7

Parallel Version (Par. PNL) • Goals – to develop scalable parallel algorithms of learning and inference for probabilistic graphics models – to implement developed parallel versions of algorithms in frames of symmetric multi-processing systems as well as in distributed systems • Algorithms – junction tree inference – loopy belief propagation – Gibbs sampling – dynamic junction tree inference – expectation maximization learning • EM Learning admits making in parallel propagation of evidence on E-step. 8

PNL High Level API PNL example PNL High example const int num. Of. Nds = 4; int num. Of. Nbrs[num. Of. Nds] = { 2, 2, 2, 2 }; int nbrs 0[] = { 1, 2 }; Bayes. Net net; int nbrs 1[] = { 0, 3 }; net. Add. Node(discrete^"Cloudy Sprinkler Rain Wet. Grass", "true false"); int nbrs 2[] = { 0, 3 }; net. Add. Arc("Cloudy", "Sprinkler Rain"); int nbrs 3[] = { 1, 2 }; net. Add. Arc("Sprinkler Rain", "Wet. Grass"); int *nbrs[] = { nbrs 0, nbrs 1, nbrs 2, nbrs 3 }; ENeighbor. Type nbrs. Types 0[] = { nt. Child, nt. Child }; ENeighbor. Type nbrs. Types 1[] = { nt. Parent, nt. Child }; net. Set. PTabular("Cloudy^true Cloudy^false", "0. 6 0. 4"); net. Set. PTabular("Sprinkler^true Sprinkler^false", "0. 1 0. 9", "Cloudy^true"); ENeighbor. Type nbrs. Types 2[] = { nt. Parent, nt. Child }; net. Set. PTabular("Sprinkler^true Sprinkler^false", "0. 5", "Cloudy^false"); ENeighbor. Type nbrs. Types 3[] = { nt. Parent, nt. Parent }; net. Set. PTabular("Wet. Grass^true Wet. Grass^false", "0. 99 0. 01", "Rain^true Sprinkler^true "); ENeighbor. Type *nbrs. Types[] = { nbrs. Types 0, nbrs. Types 1, nbrs. Types 2, nbrs. Types 3 }; net. Set. PTabular("Wet. Grass^true Wet. Grass^false", "0. 9 0. 1", "Sprinkler^true Rain^false"); p. Graph = CGraph: : Create( num. Of. Nds, num. Of. Nbrs, nbrs. Types ); net. Set. PTabular("Wet. Grass^true Wet. Grass^false", "0. 9 0. 1", "Sprinkler^false Rain^true"); node. Type. Vector node. Types; net. Set. PTabular("Wet. Grass^true Wet. Grass^false", "0. 0 1. 0", "Sprinkler^false Rain^false"); CNode. Type nt(1, 2); net. Set. PTabular("Rain^true Rain^false", "0. 3 0. 7", "Cloudy^true"); node. Types. push_back(nt); net. Set. PTabular("Rain^true Rain^false", "0. 5", "Cloudy^false"); int. Vector node. Association; node. Association. assign(num. Of. Nds, 0); CBNet* p. BNet = CBNet: : Create( num. Of. Nds, node. Types, node. Association, p. Graph ); p. BNet->Alloc. Factors(); float table 0[] = { 0. 5 f, 0. 5 f }; float table 1[] = { 0. 5 f, 0. 9 f, 0. 1 f }; float table 2[] = { 0. 8 f, 0. 2 f, 0. 8 f }; float table 3[] = { 1. 0 f, 0. 1 f, 0. 9 f, 0. 01 f, 0. 99 f }; float* table[] = { table 0, table 1, table 2, table 3 }; for( int i = 0; i < num. Of. Nds; ++i ) { p. BNet->Alloc. Factor(i); CFactor* p. Factor = p. BNet->Get. Factor(i); p. Factor->Alloc. Matrix( table[i], mat. Table ); } 9

PNL Statistics Source Code (available on Source Forge) ~ 4 Mb source code ~ 80 classes ~ 30 classes which are needed for user Documentation ~ 1 Mb friendly-written documentation in html-format Regression Testing Suite About 7000 users worldwide 10

PNL versus… 11

Applications Data-driven super resolution developed by Intel Russia Research Center team Zoomed 8 times up 12

Applications Facial emotion recognition Peter Robinson & Rana El Kaliouby (University of Cambridge) 13

Contact info http: //eng. itlab. unn. ru/? dir=139 http: //www. sourceforge. net/projects/openpnl 14