Hailuoto Workshop UNC Stat OR Object Oriented Data

Hailuoto Workshop UNC, Stat & OR Object Oriented Data Analysis, III J. S. Marron Dept. of Statistics and Operations Research, University of North Carolina 08 June 2021 1

HDLSS Space is a Weird Place, I UNC, Stat & OR Maximal Data Piling (with J. Y. Ahn) In HDLSS Binary Discrimination: There is direction where: Class +1 projections pile at one pt. Class -1 projections pile at another 2

HDLSS Space is a Weird Place, II UNC, Stat & OR Maximal Data Piling Mathematics: n Exists w. p. 1, when abs. cont. w. r. t. Lebesgue measure n Unique within subspace gend by Data n Formula very similar to FLD (pooled within cov. global cov. ) n Same as FLD when n < d 3

HDLSS Space is a Weird Place, III UNC, Stat & OR ~2 n MDP Dirns Useful for clustering? Hard Optimization… 4

HDLSS Space is a Weird Place, IV UNC, Stat & OR Parallel Directions (with X. Liu) 5

Time Series of Curves UNC, Stat & OR n Chemical Spectra, evolving over time (with J. Wendelberger & E. Kober) n Mortality curves changing in time (with Andres Alonzo) Visualization: n Similar tools, PCA & Dirns n But color according to time 6

Chemical Spectra, I UNC, Stat & OR 7

Chemical Spectra, II UNC, Stat & OR 8

Chemical Spectra, III UNC, Stat & OR 9

Chemical Spectra, IV UNC, Stat & OR 10

Chemical Spectra, V UNC, Stat & OR 11

Chemical Spectra, VI UNC, Stat & OR 12

Chemical Spectra, VII UNC, Stat & OR 13

Chemical Spectra, VIII UNC, Stat & OR 14

Demography Data UNC, Stat & OR Mortality, as a function of age n “Chance of dying”, for Males n of each 1 -year age group n Curves are years 1908 - 2002 PCA of the family of curves 15

Demography Data UNC, Stat & OR n n PCA of the family of curves for Males Babies & elderly “most mortal” (Raw) All getting better over time (Raw & PC 1) Except 1918 - Influenza Pandemic (see Color Scale) Middle age most mortal (PC 2): n n 1918 Early 1930 s - Spanish Civil War 1980 – 1994 (then better) auto wrecks Decade Rounding (several places) 16

Demography Data UNC, Stat & OR PCA for Males in Switzerland n Most aspects similar n No decade rounding (better records) n 1918 Flu – Different Color (PC 2) (see Color Scale) n No War Changes n Steady improvement until 70 s (PC 2) n When auto accidents kicked in 17

Demography Data UNC, Stat & OR Dual PCA Idea: Rows and Columns trade places Demographic Primal View: Curves are Years, Coord’s are Ages Demographic Dual View: Curves are Ages, Coord’s are Years Dual PCA View, Spanish Males 18

Demography Data UNC, Stat & OR Dual PCA View, Spanish Males n Olde people have const. mortality (raw) n But improvement for rest (raw) n Bad for 1918 (flu) & Spanish Civil War, but generally improving (mean) n Improves for ages 1 -6, then worse (PC 1) n Big Improvement for young (PC 2) (Age Color Key) 19

Discrimination for m-reps UNC, Stat & OR n. Classification n. S. n. What for Lie Groups – Symm. Spaces K. Sen & S. Joshi is “separating plane” (for SVM-DWD)? 20

Trees as Data Points, I UNC, Stat & OR Brain Blood Vessel Trees - E. Bullit & H. Wang Statistical Understanding of Population? Mean? Challenge: PCA? Very Non-Euclidean 21

Trees as Data Points, II UNC, Stat & OR n. Mean n. PCA of Tree Population: Frechét Approach on Trees (based on “tree lines”) n. Theory in Place - Implementation? 22

HDLSS Asymptotics: Simple Paradoxes, I UNC, Stat & OR For dim’al “Standard Normal” dist’n: Euclidean Distance to Origin (as ): - Data lie roughly on surface of sphere of radius - Yet origin is point of “highest density”? ? ? - Paradox resolved by: “density w. r. t. Lebesgue Measure” 23

HDLSS Asymptotics: Simple Paradoxes, II UNC, Stat & OR For dim’al “Standard Normal” dist’n: indep. of Euclidean Dist. between and (as Distance tends to non-random constant: ): Can extend to Where do they all go? ? ? (we can only perceive 3 dim’ns) 24

HDLSS Asymptotics: Simple Paradoxes, III UNC, Stat & OR For dim’al “Standard Normal” dist’n: indep. of High dim’al Angles (as ): - -“Everything is orthogonal”? ? ? - Where do they all go? ? ? (again our perceptual limitations) - Again 1 st order structure is non-random 25

HDLSS Asy’s: Geometrical Representation, I UNC, Stat & OR Assume , let Study Subspace Generated by Data a. Hyperplane through 0, of dimension b. Points are “nearly equidistant to 0”, & dist c. Within plane, can “rotate towards Unit Simplex” d. All Gaussian data sets are“near Unit Simplex Vertices”!!! “Randomness” appears only in rotation of simplex With P. Hall & A. Neemon 26

HDLSS Asy’s: Geometrical Representation, II UNC, Stat & OR Assume , let Study Hyperplane Generated by Data a. dimensional hyperplane b. Points are pairwise equidistant, dist c. Points lie at vertices of “regular hedron” d. Again “randomness in data” is only in rotation e. Surprisingly rigid structure in data? 27

HDLSS Asy’s: Geometrical Representation, III UNC, Stat & OR Simulation View: shows “rigidity after rotation” 28

HDLSS Asy’s: Geometrical Representation, III UNC, Stat & OR Straightforward Generalizations: n non-Gaussian data: only need moments n non-independent: n Mild Eigenvalue condition on Theoretical Cov. use “mixing conditions” (with J. Ahn, K. Muller & Y. Chi) All based on simple “Laws of Large Numbers” 29

HDLSS Asy’s: Geometrical Representation, IV UNC, Stat & OR Explanation of Observed (Simulation) Behavior: “everything similar for very high d” n 2 popn’s are 2 simplices (i. e. regular n-hedrons) n All are same distance from the other class n i. e. everything is a support vector n i. e. all sensible directions show “data piling” n so “sensible methods are all nearly the same” n Including 1 - NN 30

HDLSS Asy’s: Geometrical Representation, V UNC, Stat & OR Further Consequences of Geometric Representation 1. Inefficiency of DWD for uneven sample size (motivates “weighted version”, work in progress) 2. DWD more “stable” than SVM (based on “deeper limiting distributions”) (reflects intuitive idea “feeling sampling variation”) (something like “mean vs. median”) 3. 1 -NN rule inefficiency is quantified. 31

The Future of Geometrical Representation? UNC, Stat & OR n HDLSS version of “optimality” results? n “Contiguity” approach? n Rates of Convergence? n Improvements of DWD? Params depend on d? (e. g. other functions of distance than inverse) It is still early days … 32

Some Carry Away Lessons UNC, Stat & OR n Atoms of the Analysis: Object Oriented n HDLSS contexts deserve further study n DWD is attractive for HDLSS classification n “Randomness” in HDLSS data is only in rotations (Modulo rotation, have context simplex shape) n How to put HDLSS asymptotics to work? 33