Differential Deep Learning on Graphs and its Applications

Differential Deep Learning on Graphs and its Applications Chengxi Zang and Fei Wang Weill Cornell Medicine www. calvinzang. com DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 1

This Tutorial qwww. calvinzang. com/DDLG_AAAI_2020. html q. AAAI-2020 q. Friday, February 7, 2020, 2: 00 PM -6: 00 PM q. Sutton North, Hilton New York Midtown, NYC DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 2

This Tutorial q. Molecular Graph Generation: to generate novel molecules with optimized properties o. Graph generation o. Graph property prediction o. Graph optimization q. Learning Dynamics on Graphs: to predict temporal change or final states of complex systems o. Continuous-time network dynamics prediction o. Structured sequence prediction o. Node classification/regression q. Mechanism discovery: to find dynamical laws of complex systems o. Density Estimation vs. Mechanism Discovery o. Data-driven discovery of differential equations DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 3

Molecular Graph Generation q P( )? f( DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 )=? 4

Mo. Flow: An Invertible Flow Model for Generating Molecular Graphs DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 5

Learning Dynamics on Graphs q Adjacency Matrix Graph + Dynamic Process DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 Dynamics of each nodes ? 6

Neural Dynamics on Complex Networks q

Mechanism Discovery q. Goals: To find dynamical laws of complex systems q. Graph Analysis tasks o. Density estimation vs. mechanism discovery o. Data-driven discovery of differential equations ? Image from http: //networksciencebook. com/chapter/4#hubs DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 8

9 Dynamical Origins of Distribution Functions Distribution q. A theorem constructing dynamic systems described by Differential Equations which generate the observed distribution Data Survival Analysis Stochastic, linear Forward: data generation Input Deterministic, nonlinear Dynamical system DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 Dynamic System ODE Complex Pattern Output Backward: system identification/Le arning 9

Some Practical Tips q Data preprocessing o Padding null atoms, augmenting null edges q Normalization matters o Graphnorm, batchnorm, actnorm q Stable flows with less reconstruction error o Normalization, sigmoid, checking each layer q Discrete mapping is faster than integration q Split and coupling layer are very efficient invertible framework for graph convolution q Visualizing dynamics on graphs q Thinking physical meanings of differential equations CNOF* DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 10

Differential Deep Learning on Graphs q. Graphs and Differential Equations are general tools to describe structures and dynamics of complex systems q. Inspired by the Differential Equations, we can design and analyze Deep Models q. For applications on graphs (our focus), including: o. Molecular Graph Generation o. Learning dynamics of complex systems o. Mechanism discovery in a data-driven manner DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 11

More Directions q. Deep Learning Differential Equations o. Analysis v. Math analysis tools v. Concepts in dynamic system and control: stability, robustness, complexity, resilience, etc. o. Modeling Continuous-time process v. Physical meaning. The laws of nature are expressed as differential equations. q. Differential Equations Deep Learning o. Design v. There are many dynamical systems and differential equations. v. Discretization of continuous time-varying neural dynamics Deep Neural Networks v. DNNs implemented by modern auto-differentiation softwares are more flexible, expressive and efficient o. Generative models and Invertible structures DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 12

More Directions q. Applications o. Network medicine o. Drug discovery o. Molecular dynamics o. Urban computing o. Social networks o. Recommendation o. Etc. (structures + dynamics) DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 13

Thank You! Differential Deep Learning on Graphs and its Applications Chengxi Zang and Fei Wang Weill Cornell Medicine www. calvinzang. com DIFFERENTIAL DEEP LEARNING ON GRAPHS AND ITS APPLICATIONS --- AAAI-2020 14