Machine Learning and Statistical Analysis for Materials Science

Machine Learning and Statistical Analysis for Materials Science Umesh V Waghmare Materials Theory Group Theoretical Sciences Unit J Nehru Centre for Advanced Scientific Research Jakkur PO, Bangalore 560 064 INDIA http: //www. jncasr. ac. in/waghmare@jncasr. ac. in Funding from DST, Shell, SSL, IKST

History of Usage of Materials https: //depts. washington. edu/matseed/ces_guide/background. htm QM ML Natural Stone-Bronze-Iron-…semiconductor ages … IT 1950’s: A Remarkable Change, birth of Materials Science

Materials Science Materials science is a syncretic discipline hybridizing metallurgy, ceramics, solid-state physics, and chemistry. It is the first example of a new academic discipline emerging by fusion rather than fission Rustum Roy (1979) PSU Press How the history of a material (its processing) influences its structure, and thus the material's properties and performance. 2 Experiment +Theory Kinetics Atomic-nano-micro-macro 3 Interaction with fields 1 4 Materials Paradigm Quantum Physics Applications Truly exciting science that impacts society https: //en. wikipedia. org/wiki/Materials_science

Materials: Structure and Properties Renewable Energy Conversion Transparent CO e Absorber h Catalyst Energy Storage Fuels: H 2, CH 3 OH Li+ Fuel Cell Electricity Energy Consumption: Efficiency Properties: rearrangement of atoms, e Interaction between electrons, atoms, molecules, solids

First-principles Quantum Theoretical Description of a Material e Ni e e Al Ni e Ni Al e Ni A Material = Collection of nuclei and electrons that interact via electro-magnetic fields Nuclear motion: Newton’s laws Electronic motion: Wave Mechanics Lighter, faster electrons: Quantum ground state (G. S. ) Given atomic positions, Electrons distribute to remain in many body G. S. Electronic Structure! Adiabatic or Born-Oppenheimer Approximation “Total Energy” of Electrons and Nuclei: Inter-atomic Interaction Potential Electrons--glue

First-principles Theoretical Approach: Total Energy Function Chemistry: Structure: ZI: Atomic numbers of atoms in a given material RI: Atomic positions of atoms in a given material Quantum Mechanics Electrostatic Energy Minimum energy quantum state of electrons: Density Functional Theory, W Kohn et al • Inter-atomic potential T=0 K • Hamiltonian of a collection of atoms Stat Mech Nobel (Chem), 1998 Free Energy T>0 K

Total Energy Function: fundamental structure ↔ property relation Etot Bond length Cohesive energy Etot=Ec+(1/2) K (d-d 0)2 1. Et is the interatomic interaction potential: function of atomic positions [structure] and numbers [chemistry]! 2. Atomic positions that minimize Et: Structure at T=0 K 3. Differences in Et ↔ Properties of a material

Materials and Society Materials exhibit a remarkable diversity in properties that have been key to technological revolutions. The origin of diversity in the World of Materials: 1. Chemical constituents: which elements or atoms? 2. Structure of a material: how are the atoms and their electrons organized in space-time? Structure and Chemistry Science Design Materials Design: Challenging Task Properties Machine Learning?

Design of Materials First-principles Information Neural Network Materials Databases Modeling: Artificial Neural Networks Structure and Machine Learning Properties Bhadeshia, ISIJ Int (1999) Target Properties DJ Scott et al J. Chem. Inf. Model. , Genetic 48, 262 (2008). Algorithm Solving Inverse problem New Material

Introduction to Machine Learning

Materials Data M 1, M 2, … X 1 X 2. . . D 1 D 2. . . Xd Dd MN

Machine Learning Supervised Learning Develop an approximate predictive model for y=f(x) Linear regression, regularization, LASSO, neural networks, Kernel regr. Unsupervised Learning Exploratory, find patterns in data x: Grouping, Clustering Principal Component Analysis, Singular Value Decomp, Clustering Regression Classification Practice: use a combination of both!

Machine Learning X Basic Material Data f(x) An Unknown function Electron Density DFT Tolerance Ratio ? TEM Image ? Composition+Structure Synthesis Conditions (P, T) Material Property Total Energy FE /AFE ground state Shear Strength Property; e. g. Tc ? ? Structure

Linear Regression W 0 x 0 W 1 x 1 W 2 x 2 ……. Model Parameters WN x. N DATA: k material

Linear Regression (contd) Parameters wi’s define the model [Least Square] Error Function (to be minimized) Parameters wi’s minimize the error What if the matrix x. Tx is singular? • Principal Component Analysis / SVD (Unsupervised Learning) • Regularization: LASSO, Ridge and Elastic Ridge Regressions

Regularization LASSO: Least Absolute Shrinkage Selection Operator regression with L 1 regularization > LASSO results in sparse models with fewer coefficients: some of the ‘s shrink to zero, giving a simple and interpretable model Ridge Regression: regression with L 2 regularization Elastic Net Regression: regression with L 2 and L 1 regularization

Principal Component Analysis 1. 2. 3. 4. 5. Start with the matrix of data X of size (d*M) Subtract the average of x from each dimension Calculate the covariance matrix A=XTX of data points. Calculate eigen vectors and corresponding eigen values of A. Sort the eigen vectors according to their eigen values in decreasing order. 6. Choose first k eigen vectors and that will be the new k dimensions. 7. Transform the original d dimensional data points into k dimensions. Remove the data in redundant dimensions, Retain only the k important dimensions [principal components] Similar goals can be accomplished using Singular Value Decomposition of A. Here, non-zero singular values identify the k important dimensions.

Kernel Regression: Parameter-free! A Smoothing Kernel function A Kernel regression is a weighted average! 1. The only parameter to be tuned is h! 2. Issues at the Boundary of the domain of x

Logistical Regression: Perceptron σ Sigmoid x W 0 x 0 W 1 x 1 W 2 Model Parameters WN x 2 x. N A Taspinar DATA: k material

Deep Learning Neural Network W http: //www. opennn. net/

Artificial Neural Network (ANN) A large vector of data is inputs and output is to match properties of interest, both available from databases. y 2 Training: Determination of wij’s from a set of inputs and outputs, and a choice of f

Data-intensive (Knowledge-based) Approach In Materials Science Two Examples

Machine Learning for Interatomic Potential (Etot) J Behler, J Chem Phys 145, 170901 (2016) Bulk Silicon Behler and Parrinello, Phys Rev Lett 98, 146401 (2007) • Structural Descriptors based on neighbor density and its symmetry-respecting expansion in sphetical harmonics • Limitation when more than a few types of atoms

Crystal Graph Convolutional Neural Networks Training with 20, 000 to 200, 000 Data points In Mat Databases Park & Wolverton, ar. Xiv. 19060526 Xie and Grossmann, Phys Rev Lett 120, 145301 (2018). Weight and Bias Concatenation of nodes (i, j) and edge vector (k) Very Successful in modeling formation energy, gaps, moduli, etc

Obtaining insights into complex phenomena: Structural Features relevant to glassy dynamics Schoenholz, Cubuk, Sussman, Kaxiras and Liu, Nature Phys 12, 469 (2016). Is structure important to glassy dynamics? Local structural environment: 166 str functions Use Support Vector Machines method to identify soft and hard particles Soft: particles that rearrange during dynamics SVM: T-evolution of softness glassy dynamics

Data-intensive (Knowledge-based) Approach In Design of Materials

Focus on Functional materials, not Structural Materials Ceder (Structure of alloys) 2003 Ceder (Materials Genome-Li batteries) 2006 Curtarolo (AFLOWlib: alloys) 2006, 2012 Magnetic Heuslers J Noskov (SUNCAT: catalysts) 2006 Persson-Ceder (Materials Project) 2011 Wolverton (OQMD: perovskites) 2013 Uof. Minn (Nanoporous Mat. Genome. Center) Marzari (MARVEL-Materials Cloud: 2 -D M) 2016 Smit (Algorithms: MOFs) Nicola Nosengo, Nature 533, 22 (2016).

Machine Learning in Materials Science The Material Code: Machine-learning techniques could revolutionize how materials science is done Nicola Nosengo, Nature 533, 22 (2016). No computationally predicted material in the market yet. Combining experiment and theory is essential Need: Methods that integrate synthesis, micro-structure, properties and performance

HTE: high throughput experiments

Knowledge(Data)-based Approach ü It is very general, has been used in prediction of ceramic materials, alloys and optimizing processing conditions. ü Various properties can be integrated together in the process of design.

High throughput screening of materials requires (a) frequent calculation of the inputs to the M/L model of properties (b) M/L model to generalize and work for other classes of materials

Artificial Neural Networks 1. Need large data; goal is accuracy of fitting 2. Artificial Neural Networks are like black box: Insights hard to obtain Being nonlinear, the model is not unique 3. Accuracy depends on the accuracy of data: interpolation; totally new materials? Inverse Problem: High throughput screening of materials 1. Need to calculate Property=f(data) frequently 2. f(data) and data should be quick calculation Descriptors The data that is 1. Relevant to desired property 2. possibly insightful 3. “minimal set” 4. Easy to compute

In the remaining of the talk, I present two approaches to determine descriptors 1. Intuitive Approach 2. Machine Learning Approach

Descriptors of Site-specific Catalytic Activity of B and N substituted Graphene Catalysis of Oxygen Reduction Reaction

Fuel Cells Generate Electricity from a fuel (e. g. Hydrogen) Oxygen Reduction Reaction at the Cathode (Pt) Need for a material to replace Pt catalyst

Descriptors of Catalytic Activity Noskov et al Descriptors: Easy to calculate Correlate with Activity Electronic states near EF or Gap Sinthika, Waghmare and Thapa (2018)

Various Substitutional Sites of C 1 -x(BN)x for Catalysis DATA S O M P Ortho Para Meta At e. F, pz orbital states are prominent: π Electronic Structure

Descriptors of ORR Activity of Sites of C 1 -x(BN)x and Relevant Free Energy ΔGOH is the “Key Performance Index”

A single LOCAL Electronic Descriptor of C 1 -x(BN)x Knowledge of projected density of electronic states: Site-specific catalytic activity towards ORR

Structural Descriptors of ORR activity of C 1 -x(BN)x S O M P Predictive Model: Easy to use, generalize to other π–bonded 2 -D materials

Most Optimal Site C 1 -x(BN)x for ORR N N N 3 N-para site: lowest over-potential of 0. 48 e. V Big Data Descriptors Chemical Intuition Predictive Model Design of Materials Insights Sinthika, Waghmare and Thapa, Small 14, 1703609 (2018)

Determination of Descriptors (and predictive models) Using Machine Learning An integrated frame-work for modeling!

Our Approach Generalize the notion of descriptors: Start with a large number of descriptors (space of D’s) M 1, M 2, … D 1 D 2. . . DM MN D 1. Df M 1, M 2, … MN Insights: Minimal set, Fingerprint Descriptors

Idea A minimal set ANN Model: y=f(D) Our Model: y=f(X=Df) 44

Advantages of our Recipe based on Bo. PGD Algorithm (over other schemes, like ANN, LASSO) Ceder et al Scheffler et al Our Machine Learning recipe: 1. Work well with small data-sets too 2. It scales well with data-size (uses clustering) 3. More than accuracy, focus on insight/mechanism 4. Finger-print Descriptors: Unique, minimal set stability of this set gives transferability of the model Finding the right set of features to represent data is the key to learning! Pankajakshan, Sanyal, de Noord, Bhattacharya, Bhattacharyya, Waghmare, Chem of Mat 29, 4190 (2017)

(f-descriptors) Clustering

The Recipe: From First-principles To Fingerprint Descriptors To Model Bootstrapped Projected Gradient Descent Algorithm (Bo. PGD) The Model: Chemical Insights & Predictive 47

Catalysts for CO 2 Reduction Use the published database: 1. ~ 300 materials (metals and alloys) 2. 13 descriptors 3. Key Performance Index (KPI)=binding energy of CO Reference (used ANN):

Preprocessing: Handle missing data, Normalize data Remove outliers: Violin Plots Distribution Descriptors

Preprocessing (continued): Examine correlations +ve KPY Heatmap: Pearson’s product moment correlation Graph Plot Correlations between descriptors: rationalized with physics/chemistry → clustering of descriptors e. g. matrix element VAD 2 and Width of d-band Clustering of descriptors in Bo. PGD algorithm

P engineered Descriptors Pearson Correlation, Graphs Small Data Clustering of Descriptors (C<P) Training and Testing Sets Bootstrapping (T): Stability Projected Gradient Descent: Sparse LR KPY=β. Xc Rank the descriptors from Tx. K (< C) Fingerprint Descriptors: Model Representation Supervised Learning with Sparsity

Catalysts for CO 2 reduction Reference: Use the published database: 1. ~ 300 materials 2. 13 descriptors 3. Key Performance Index (KPI)=binding energy of CO X X X d-band Model X 1. Predictive Model 2. Chemical Insights Additional Descriptor

Mechanism: Catalysis of CO 2 Reduction Identified the Ingredients of d-band model (Noskov et al) Pankajakshan, Sanyal, de Noord, Bhattacharya, Bhattacharyya, Waghmare, Chem of Mat 29, 4190 (2017) 53

Our machine learning scheme: Not only help derive the d-band model by learning from the data, but also proposed an additional ingredient (W) to improve it!

Model for dielectric break-down field (Fb) Capacitive Energy Storage Use published database: C. Kim, G. Pilania, R. Ramaprasad, Chem. Mater. , 2016, 28 (5), pp 1304– 1311. Descriptor 1 2 3 4 5 6 7 8 Band Gap Phonon cut off frequency Average Phonon Frequency electronic part of the dielectric constant total dielectric constant nearest neighbor distance Mass Density Bulk Modulus Notati on Eg ɷmax ɷmean ɛe Value Range ɛtot d. NN ρ M 4. 17 -57. 213 1. 523 -3. 604 2. 317 -10. 251 18. 317 -460. 524 LASSO-model • • Physical interpretation hard Descriptor ωmax hard to compute 0. 2 -13. 6 2. 914 -40. 513 1. 415 -29. 674 1. 821 -26. 29

Our Analysis Preprocessing of Data: Outliers and Clusters Pearson Correlation Principal Component Analysis Euclidean distance based clustering of materials

Machine Learning Constrained by Dimensional Analysis and Scaling Laws Learn from Wisdom, Existing Laws Kumar, et al, Chem of Materials 31, 314 (2019).

A B Using (A) Dimensional Analysis & (B) Scaling Laws Buckingham Pi Theorem Fb, Eg, wmax, B, εe Quantum Perturbation Theory Integrate this analysis with Machine Learning

Non-unique Models of Breakdown Field from Machine Learning Ramprasad et al, Chem. Mater. , 2016, 28 (5), pp 1304– 1311. BOPGD++: Machine learning + Physical Laws Our Model (Eqn 3): Simple, depends on quantities easy to calculate Physically Interpretable

Our Scheme: Models easy to interpret and compute! C. Kim et al, Chem. Mater. , 2016, 28 (5), pp 1304– 1311. Our analysis =ΔV Band-gap (Eg) and dnn(distance between cation and anion): Easy to measure or compute! Physical Meaning: Change in the electrostatic potential ΔV = Eg Smaller Gap, Far separated ions make it easy for Dielectric Breakdown!

Evidence for Transferability of our Model Our model having simple form, with physics in it: does better for totally different compounds and crystal structure

Learning from Data and Wisdom Knowledge-based approach to design materials, based on a combination of 1. Machine Learning 2. Dimensional Analysis, Physical & Scaling Laws Simple, Interpretable &Transferable Predictive Model of Functionality Kumar, et al, Chem of Materials 31, 314 (2019) High throughput screening & design of materials Machine Learning: Holistic Integration of models of processing, structure, property, performance of a material

Collaborators Narendra Kumar JNCASR Sinthika & R Thapa, SRM University P Pankajakshan, P Rajagopalan, J Balachandran & Suchismita Sanyal (Shell) I Bhattacharya & Arnab Bhattacharyya (IISc)

Thank You!

Materials for aerospace: Ashby charts Weight = 650 tonnes Al-based composites with Carbon fibre reinforced plastics Materials cluster together: patterns in the data

Materials Design: ANN & GA - Issues of small data sets and extrapolation - accuracy versus transferability - Idea of descriptors -- direct intuitive/mechanistic approach -- Machine Learning approach * DFT * Str-Prop ICME talk TSU Additional slides - Intuitive approach: graphene based catalysts for ORR Japan - Bo. PGD scheme: d-band model! Shell-9 - New slides Dimensional Analysis + Machine Learning Take-home messages: Generalized descriptors, Finger-print descriptors Intuitive scheme Machine Learning schemes Dependence on M/L algorithm!