Technische Universitt Mnchen Sensor Fusion with Deep Learning

Technische Universität München Sensor Fusion with Deep Learning Özgür Akyazı 17. 01. 2019 Kickoff: M. Sc. Data Engineering and Analytics Supervisor(s): Adnane Jadid

Fachgebiet Augmented Reality Technische Universität München Index Introduction/Motivation Problem Description: Issues Goals of this Project Existing Solutions Approach / Related Work Neural Network(NN) Design Discussion of Potential Issues Time Line References Akyazı: Sensor Fusion with Deep Learning 2

Fachgebiet Augmented Reality Technische Universität München Introduction / Motivation Tracking is one of the main tasks in Augmented Reality. Should be precise, accurate, and robust, as possible. Need multiple sensors → Sensor Fusion Deep Learning is a very trending topic !! Akyazı: Sensor Fusion with Deep Learning 3

Fachgebiet Augmented Reality Technische Universität München Problem Description: Issues Using 2 or more IMU sensors, realize sensor fusion to get the position, orientation, acceleration and angular velocity of a target. Existing solutions for sensor fusion are almost all analytical approaches. These methods have a drawback of sensor registration. State of the art solutions require very deep mathematical knowledge Akyazı: Sensor Fusion with Deep Learning 4

Fachgebiet Augmented Reality Technische Universität München Problem Description: Issues IMU’s are very noisy! Akyazı: Sensor Fusion with Deep Learning 5

Fachgebiet Augmented Reality Technische Universität München Goals of this Project Akyazı: Sensor Fusion with Deep Learning 6

Fachgebiet Augmented Reality Technische Universität München Existing Solutions Bayesian Inference Kalman Filter Well established Appropriate only for linear systems Extended/Unscented Kalman Filter Appropriate also for nonlinear systems Achieves this by approximating a nonlinear function using Taylor series Federated Kalman Filter Moving Horizon Estimation Online optimization based approach Stochastic non-linear state estimator Fuzzy Logic Continuous valued truth values Akyazı: Sensor Fusion with Deep Learning 7

Fachgebiet Augmented Reality Technische Universität München Approach Convolution Operation Source: [3] Source: TUM Augmented Reality Lecture Akyazı: Sensor Fusion with Deep Learning 8

Fachgebiet Augmented Reality Technische Universität München Approach Convolutional Layer Feature Extraction Akyazı: Sensor Fusion with Deep Learning 9

Fachgebiet Augmented Reality Technische Universität München Approach Recurrent Neural Network(RNN) Extracting time(sequential) dependencies in the input Akyazı: Sensor Fusion with Deep Learning 10

Fachgebiet Augmented Reality Technische Universität München Approach Long Short-Term Memory(LSTM) Improved RNN by Sepp Hochreiter and Jürgen Schmidhuber Akyazı: Sensor Fusion with Deep Learning 11

Fachgebiet Augmented Reality Technische Universität München Related Work Example Architecture From Literature Used for Human Activity Recognition(HAR)[1] Akyazı: Sensor Fusion with Deep Learning 12

Fachgebiet Augmented Reality Technische Universität München Related Work Camera Fused with IMU using Deep Learning Closest study in the Literature[2] Akyazı: Sensor Fusion with Deep Learning 13

Fachgebiet Augmented Reality Technische Universität München Neural Network(NN) Design Akyazı: Sensor Fusion with Deep Learning 14

Fachgebiet Augmented Reality Technische Universität München Discussion of Potential Issues Data Collection with use of Ubi. Track, non-synchronized IMU measurement 3 different IMUs are used Frequency difference in sensors Different APIs Different precisions Long training time, because of Complex NN architecture Hyperparameter search Learning Rate Dropout probabilities Number of units in each layer Akyazı: Sensor Fusion with Deep Learning 15

Fachgebiet Augmented Reality Technische Universität München Time Line 15. 10. 2018 - 14. 12. 2018 Literature Research, Review and writing the Motivation and Related Work parts of the Final Report 15. 12. 2018 - 14. 02. 2019 Data Collection, Design, Implementation and Training of the NN, Writing Implementation part of the report 15. 02. 2019 - 14. 03. 2019 Evaluation and writing of the Evaluation part of the report 15. 03. 2019 - 30. 03. 2019 Writing the Conclusion and Future Work parts of the report Akyazı: Sensor Fusion with Deep Learning 16

Fachgebiet Augmented Reality Technische Universität München List of References 1. Ordóñez, F. J. , & Roggen, D. (2016). Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors (Switzerland). 2. Rambach, J. R. , Tewari, A. , Pagani, A. , & Stricker, D. (2016). Learning to Fuse: A Deep Learning Approach to Visual-Inertial Camera Pose Estimation. 3. https: //developer. apple. com/library/archive/documentation/Performance/Con ceptual/v. Image/Convolution. Operations. html Akyazı: Sensor Fusion with Deep Learning 17

Fachgebiet Augmented Reality Technische Universität München Thank You. . . Akyazı: Sensor Fusion with Deep Learning 18