Sufficient Markov Decision Processes with Alternating Deep Neural

Sufficient Markov Decision Processes with Alternating Deep Neural Networks Marshall Wang (lwang 31@ncsu. edu) Dept. of Statistics, NC State University Advisor: Eric Laber Aug 4, 2017 1

Motivation • Want to apply mobile intervention to students with heavy drinking/smoking behavior • Hard to identify an optimal strategy when data are high dimensional and noisy • Need a dimension reduction that retains useful information 2

Contributions • Provided a criterion to measure the quality of a dimension reduction • Designed a deep learning model to produce a dimension reduction with no information loss • Demonstrated the method on a mobile intervention study 3

Outline Ø Sufficient Markov Decision Process Ø Alternating Deep Neural Network Ø Simulation Study Ø Application on Mobile Health 4

Ø Sufficient Markov Decision Process Ø Alternating Deep Neural Network Ø Simulation Study Ø Application on Mobile Health 5

Markov Decision Process 6

Markov Decision Process 7

Sufficient Markov Decision Process 8

Ø Sufficient Markov Decision Process Ø Alternating Deep Neural Network Ø Simulation Study Ø Application on Mobile Health 9

Deep Neural Networks (DNN) 10

Naive Dimension Reduction with DNN • Recall our criterion for a sufficient dimension reduction: 11

Alternating Deep Neural Networks • Recall our criterion for a sufficient dimension reduction: 12

Ø Sufficient Markov Decision Process Ø Alternating Deep Neural Network Ø Simulation Study Ø Application on Mobile Health 13

Setup • Additionally, add 200 noise variables including constants, white 14 noise, and dependent noise.

Results 15

Ø Sufficient Markov Decision Process Ø Alternating Deep Neural Network Ø Simulation Study Ø Application on Mobile Health 16

Data • BASICS-Mobile is a mobile intervention targeting heavy drinking and smoking among college students. • Enrolled 30 students and lasted for 14 days. • On each afternoon and evening, the student is asked to complete a list of self-report questions. Then either an informational module or a treatment module is provided. • 15 variables are collected, including baseline information, answers to self-report questions, weekend indicator and so on. • We’ll focus on smoking: Find an optimal intervention strategy to minimize cigarettes smoked. 17

Dimension Reduction with ADNN 18

Dimension Reduction with ADNN 19

Questions? Main References: 1. 2. 3. 4. 5. 6. Learning Deep architectures for AI, Y. Bengio, 2009. Reducing the dimensionality of data with neural networks, G. E. Hinton & R. Salakhutdinov, 2006. Neural networks and deep learning, M. A. Nielsen, 2015. Reinforcement learning: an introduction, R. Sutton & A. Barto. Brownian distance covariance, G. Szekely & M. Rizzo. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion, P. Vincent et al. 7. Conditional distance correlation, X. Wang et al. 8. Development and evaluation of a mobile intervention for heavy drinking and smoking among college students, K. Witkiewitz et al. Thank you! lwang 31@ncsu. edu 20