SPARSE TENSORS DECOMPOSITION SOFTWARE Papa S Diaw Masters

SPARSE TENSORS DECOMPOSITION SOFTWARE Papa S. Diaw, Master’s Candidate Dr. Michael W. Berry, Major Professor 12/3/2020 1

Introduction • Large data sets • Nonnegative Matrix Factorization (NMF) § Insights on the hidden relationships § Arrange multi-way data into a matrix • • 12/3/2020 Computation memory and higher CPU Linear relationships in the matrix representation Failure to capture important structure information Slower or less accurate calculations 2

Introduction (cont'd) • Nonnegative Tensor Factorizations (NTF) § Natural way for high dimensionality § Original multi-way structure of the data § Image processing, text mining 12/3/2020 3

Tensor Toolbox For MATLAB • Sandia National Laboratories • Licenses • Proprietary Software 12/3/2020 4

Motivation of the PILOT • Python Software for NTF • Alternative to Tensor Toolbox for MATLAB • Incorporation into Future. Lens • Exposure to NTF • Interest in the open source community 12/3/2020 5

Tensors • • • Multi-way array Order/Mode/Ways High-order Fiber Slice Unfolding § Matricization or flattening § Reordering the elements of an N-th order tensor into a matrix. § Not unique 12/3/2020 6

Tensors (cont’d) • Kronecker Product • Khatri-Rao product § A⊙B=[a 1⊗b 1 a 2⊗b 2… a. J⊗b. J] 12/3/2020 7

Tensor Factorization • Hitchcock in 1927 and later developed by Cattell in 1944 and Tucker in 1966 • Rewrite a given tensor as a finite sum of lowerrank tensors. • Tucker and PARAFAC • Rank Approximation is a problem 12/3/2020 8

PARAFAC • Parallel Factor Analysis • Canonical Decomposition (CANDE-COMPE) • Harsman, Carroll and Chang, 1970 12/3/2020 9

PARAFAC (cont’d) • Given a three-way tensor X and an approximation rank R, we define the factor matrices as the combination of the vectors from the rank-one components. 12/3/2020 10

PARAFAC (cont’d) 12/3/2020 11

PARAFAC (cont’d) • Alternating Least Square (ALS) • We cycle “over all the factor matrices and performs a least-square update for one factor matrix while holding all the others constant. ”[7] • NTF can be considered an extension of the PARAFAC model with the constraint of nonnegativity 12/3/2020 12

Python • Object-oriented, Interpreted • Runs on all systems • Flat learning curve • Supports object methods (everything is an object in Python) 12/3/2020 13

Python (cont’d) • Recent interest in the scientific community • Several scientific computing packages § Numpy § Scipy • Python is extensible 12/3/2020 14

Data Structures • Dictionary § Store the tensor data § Mutable type of container that can store any number of Python objects § Pairs of keys and their corresponding values • Suitable for sparseness of our tensors • VAST 2007 contest data 1, 385, 205, 184 elements, with 1, 184, 139 nz • Stores the nonzero elements and keeps track of the zeros by using the default value of the dictionary 12/3/2020 15

Data Structures (cont’d) • Numpy Arrays § Fundamental package for scientific computing in Python § Khatri-Rao products or tensors multiplications § Speed 12/3/2020 16

Modules 12/3/2020 17

Modules (cont’d) • SPTENSOR § Most important module § Class (subscripts of nz, values) • Flexibility (Numpy Arrays, Numpy Matrix, Python Lists) § Dictionary § Keeps a few instances variables • Size • Number of dimensions • Frobenius norm (Euclidean Norm) 12/3/2020 18

Modules (cont’d) • PARAFAC § coordinates the NTF § Implementation of ALS § Convergence or the maximum number of iterations § Factor matrices are turned into a Kruskal Tensor 12/3/2020 19

Modules (cont’d) 12/3/2020 20

Modules (cont’d) 12/3/2020 21

Modules (cont’d) • INNERPROD § Inner product between SPTENSOR and KTENSOR § PARAFAC to compute the residual norm § Kronecker product for matrices • TTV § Product sparse tensor with a (column) vector § Returns a tensor • Workhorse of our software package • Most computation • It is called by the MTTKRP and INNERPROD modules 12/3/2020 22

Modules (cont’d) • MTTKRP § Khatri-Rao product off all factor matrices except the one being updated § Matrix multiplication of the matricized tensor with KR product obtained above • Ktensor § Kruskal tensor § Object returned after the factorization is done and the factor matrices are normalized. § Class • Instance variables such as the Norm. • Norm of ktensor plays a big part in determining the residual norm in the PARAFAC module. 12/3/2020 23

Performance • Python Profiler § Run time performance § Tool for detecting bottlenecks § Code optimization • negligible improvement • efficiency loss in some modules 12/3/2020 24

Performance (cnt’d) • Lists and Recursions 12/3/2020 25

Performance (cnt’d) • Numpy Arrays 12/3/2020 26

Performance (cnt’d) • After removing Recursions 12/3/2020 27

Floating-Point Arithmetic • Binary floating-point § “Binary floating-point cannot exactly represent decimal fractions, so if binary floating-point is used it is not possible to guarantee that results will be the same as those using decimal arithmetic. ”[12] § Makes the iterations volatile 12/3/2020 28

Convergence Issues 12/3/2020 29

Convergence Issues (ctn’d) 12/3/2020 30

Convergence Issues (cont’d) 12/3/2020 31

Conclusion • There is still work to do after NTF § Preprocessing of data § Post Processing of results such as Future. Lens • Expertise • Extract and Identify hidden components • Tucker Implementation. • C extension to increase speed. • GUI 12/3/2020 32

Acknowledgments • Mr. Andrey Puretskiy § Discussions at all stages of the PILOT § Consultancy in text mining § Testing • Tensor Toolbox For MATLAB (Bader and Kolda) § Understanding of tensor Decomposition § PARAFAC 12/3/2020 33

References 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12) http: //csmr. ca. sandia. gov/~tgkolda/Tensor. Toolbox/ Tamara G. Kolda, Brett W. Bader , “Tensor Decompostions and Applications”, SIAM Review , June 10, 2008. Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, Shun-ichi Amari, “Nonnegative Matrix and Tensor Factorizations”, John Wiley & Sons, Ltd, 1009. http: //docs. python. org/library/profile. html http: //www. mathworks. com/access/helpdesk/help/techdoc http: //www. scipy. org/Num. Py_for_Matlab_Users Brett W. Bader, Andrey A. Puretskiy, Michael W. Berry, “Scenario Discovery Using Nonnegative Tensor Factorization”, J. Ruiz-Schulcloper and W. G. Kropatsch (Eds. ): CIARP 2008, LNCS 5197, pp. 791 -805, 2008 http: //docs. scipy. org/doc/numpy/user/whatisnumpy. html Tamara G. Kolda, “Multilinear operators for higher-order decompositions”, SANDIA REPORT, April 2006 http: //speleotrove. com/decimal/decifaq 1. html#inexact 12/3/2020 34

QUESTIONS? 12/3/2020 35