Data Mining in Robotics DeliaAlexandrina Mitrea Technical University

Data Mining in Robotics Delia-Alexandrina Mitrea Technical University of Cluj-Napoca

§ Topics § What is data-mining? § Data Mining Methods § Data Mining in Robot based Applications • Main Directions • Specific Methods for PLM § Specific Tools for Data Mining § Conclusions § References

What is data-mining? Ø Data Mining and Visualization § Data mining, or knowledge discovery the process of identifying new patterns and insights in data [1] § Making sense of all the data § Typical applications [1]: • understanding the Human Genome to develop new drugs • discovering new patterns in recent Census (population) data to warn about hidden trends • understanding your customers better at an electronic web-store in order to provide a personalized one-to-one experience

§ Data mining - at the intersection of multiple research areas: Machine Learning, Statistics, Pattern Recognition, Databases, and Visualization [1] § Two main goals: • Insights (descriptive data mining): identify patterns and trends that are comprehensible, so that action can be taken based on the insight Ex. : Characterize the heavy spenders on a web site, or people that buy product X. By understanding the underlying patterns => the web site can be personalized and improved • Prediction: build a model that predicts or scores, based on input data Ex. : Predict the likelihood of a certain customer to buy product X, based on the demographic data from a web-site o Types: - discrete variables with few values => classification - continuous variables => regression

Data Mining Methods Ø Data Visualization [1] • One way to aid users in understanding the models - to visualize them - Mine. Set - a data mining tool that integrates data mining and visualization very tightly - Visualization of a Naïve Bayes classifier – a set of important attributes that influence the salary level Ex. : - higher education levels (right bars in the education row) imply higher salaries because the bars are higher - salary increases with age up to a point and then decreases

Ø Other Methods § Preprocessing: feature selection (filters and wrappers); feature extraction (PCA, k. PCA, LDA, MDA) [7] § Parametric models – limited power E. g. – linear regression; Bayesian classifiers § Non-parametric models - shown to learn "any reasonable" target concept given enough data E. g. - nearest-neighbour algorithms with a growing neighbourhood; decision trees; artificial neural networks o Supervised/Unsupervised learning § Multiple model learning • Bagging - generates bootstrap samples by repeatedly sampling the training set with replacement. A model is built for each sample and they are then uniformly voted. • Boosting (Ada. Boost) - generates a set of classifiers in sequence; each classifier is given a training set where examples are reweighted to highlight those previously misclassified

§ Deep learning - extract high-level, complex abstractions as data representations through a hierarchical learning process; - analysis and learning of massive amounts of data, making it a valuable tool for Big Data Analytics; based on neural networks - both unsupervised (=> data clustering) and supervised learning (=> prediction) [2] § Associations – e. g. : find combinations of products that when bought together imply the purchase of another product • Minimum support - percentage of the data that has to satisfy the rule • Minimum confidence - the probability that the right hand side is satisfied given the lefthand side § Probably Approximately Correct (PAC) Learning - given hypothesis space (e. g. , disjunctions of conjunctions of length k), a PAC learning algorithm can learn the approximate target with high probability. - a weak learning algorithm, which can classify more accurately than random guessing (e. g. , epsilon < 0. 5), can always be boosted into a strong learning algorithm Ø Assessment: Bias/Variance decompositions • The expected error of any learning algorithm for a given target concept and training set size can be decomposed into two terms: the bias and the variance o Bias - how closely the learning algorithm’s average guess (over all possible training sets of the given training set size) matches the target o Variance - how much the learning algorithm’s guess varies for different training sets of the given size

Ø Challenges § Make data mining models comprehensible to non-experimented users - users need to understand the results of data mining § Make data transformations and model building accessible to non-experimented users - translate user’s questions referring to a data mining problem in relational format (e. g. – SQL quries) § Scale algorithms to large volumes of data. § Close the loop: identify causality, suggest actions, and measure their effect - discoveries may reveal correlations that are not causal (e. g. : human reading ability correlates with shoe size) => perform controlled experiments and measure their effect § Cope with privacy issues - data collection can also lead to abuses of the data, raising with many social and economic issues [1]

Data Mining in Robot-based Applications Ø Behaviour mining for spatial collision avoidance in multi-robot systems – gaming applications [3] Ø Human-robotics interactions through transfer of information – human queries are recognized with the help of data mining techniques [4] Ø Data mining in Product Lifecycle Management (PLM) context [5], [6], [7], [8], [9] • Big Data in PLM context [5] • Data mining driven manufacturing process optimization [7] • Multi-objective optimization and data mining => innovative design and analysis of production systems [6] • PLM-based data analytics approach for improving product development time [8] • Predictive maintenance methods for products based on big data analysis [9]

Ø Data Mining in PLM Context Ø Big data in PLM Context [5] “Big Data” a term for any collection of large and complex data sets which are difficult to be analysed by traditional data processing methods - big volume, heterogeneous, high speed of data processing, veracity (data quality) • • Data in PLM o PLM - enables companies to manage their products across their Lifecycles - manage the knowledge intensive process consisting of market analysis , product design and process development, product manufacturing, product distribution, product in use, post-sale service, and product recycling

o Phases of PLM: • Beginning of Life (BOL) – design and production • Middle of Life (MOL) - as products have existed in its final form => issues concerned with • service have become more significant and needed to be paid great attention End of Life (EOL) - product enters its EOL period => volume decisions should be made which concern the EOL product recycle or disposal o Data in BOL, MOL, EOL • Data in BOL – Marketing analysis & Product design => figuring out who are promising customers, discerning customers’ need for product - Production phase => procurement, product manufacturing, and equipment management => identifying the needs of the specific functions, making the final decisions on the details of the products, choosing qualified suppliers, monitoring product quality, simulation and testing of product, etc.

• Data in MOL, BOL and EOL

• Data in MOL - Warehouse managing => order process, inventory management, green transport planning - Customer service - Product support - Corrective maintenance - keeping and improving system availability and safety, as well as product quality - Predictive and preventive maintenance - different from corrective maintenance because actions will be taken to prevent the failure before it actually occurs => fault detection and degradation monitoring E. g. : recognize the degradation pattern - material aging mechanism in specific environment • Data in EOL - EOL product recovery decision => predicting remaining lifetime of parts or components product recovery optimization, enhancing the resource-saving recycling activities

Ø Multi-objective optimization and data mining for the innovative design and analysis of production systems [6] • Proposed approach: • Multi-objective optimization problem (MOP)

§ Knowledge discovery in Multi-Objective Optimization (MOO) • Interactive data-mining - the presence of two different spaces, objective space and decision space, adds a certain degree of difficulty in discovering knowledge of relevance - develop data mining methods that operate in the decision space, but at the same time, take the structure of solutions in the objective space into consideration - interactive data mining in MOO - allows the DM to select and analyse preferred solutions in z in the objective space to find the pattern of x*

§ Flexible Pattern Mining (FPM) • FPM allows the analysis of any z, including solutions that are not on the EF • E. g: 3 objective optimization problem • INV-TH plot + NSGA-II algorithm (clustering) => in order to generate the rules • Preferred solutions:

Ø An operation stage in the conceptual design of an automotive machining line showing alternative options: one specialized station M 1 or two standardized stations (M 1 A and M 11 A) Ø The generated rules:

Ø Data Mining Driven Manufacturing Process Optimization [7] § Data Mining in Manufacturing – 2 main directions: • Indication based manufacturing optimization (Ib. MO) - uses pre-configured manufacturing-specific data mining models to explain and predict certain process attributes • Pattern based manufacturing optimization (Pb. MO) - based on the idea of pattern-based optimization; uses manufacturing-specific optimization patterns stored in the Manufacturing Pattern Catalogue § Indication based manufacturing optimization (Ib. MO) • Conceptual use-case

o Root cause analysis (RCA) - a targeted explication use case => the data mining-based analysis of selected process characteristics defined by the user to provide comprehensible and interpretable explication models (e. g. decision trees). • Metric-oriented RCA - aims at explaining categorized metrics of process in-stances - metrics are categorized because typically only certain ranges not single values are relevant for RCA • Failure-oriented RCA - operates across the overall manufacturing process comprising all production steps to cross correlate all influence factors o Structure analysis - automatic identification of typical executions of a selected process to infer influence factors and circumstances • Cluster-oriented structure analysis => the segmentation of instances of a selected process to identify groups of typical process executions o Prediction - focuses on the forecast of certain process characteristics • Ex-ante prediction - comprises the forecast of characteristics of processes before their first execution • Real-time prediction - forecasting of process characteristics during the actual execution of the process

§ Selection of the data mining technique • The metric-oriented RCA - based on classification techniques - employs a training phase with the categorized metric as a class label • Criteria: - interpretability of the generated models from a user point of view - technical robustness • Comparison of the data mining techniques for classification:

§ Prototypical Implementation • Technical layers for root cause analysis Ø Denormalized step: execution data are merged at the level of the whole process to get one tuple per process execution Ø Data filtering => the reduction of attributes used for decision tree induction - feature selection through specific methods => focus on core attributes that significantly influence the value of the class attribute

Ø Decision tree of a metric oriented root cause analysis: => not to use machines older than 3 years in step 1 to avoid high lead times

Ø Predictive maintenance of products based on big data analysis [9] § Challenges: • Lacking of timely and accurate data of products, and lacking of useful pattern and knowledge of product lifecycle => Data flow model of product predictive maintenance through Io. T based on big data mining • • • PEID – product embedded information devices RFID - radio frequency identification PDKM - product data and knowledge management EIS – Enterprise Information System DSS- Decision Support System

§ The decision for the predictive maintenance is mainly based on the abnormal events for the product such as abnormal temperatures, abnormal vibrations • diagnosed, estimated and mined using specific prognostic or classification algorithms: decision trees, k-means clustering, support vector machine (SVM), Apriori, k. NN, Naive Bayes, Artificial Neural Networks § Data and knowledge are not only useful for corrective maintenance and predictive maintenance in the MOL phase, but also valuable for the BOL and EOL of product lifecycle • decision trees, rough set theory, ANN, SVM, association rules and other hybrid approaches => discover the knowledge and rules of MOL phase • BOL phase - designers and production engineers will receive feedback about detailed usage status data and knowledge of product from maintenance/service engineers and customers • the knowledge can be used also in EOL - product logistics or waste stream management, and product recovery decision making such as reuse, remanufacturing, material recycle and disposal and so on

Ø Deep learning • Employed mainly for prediction [11], [12] Ø Sleep quality prediction based on the physical activity recorder over the day [11] § Use specialized wearable sensors in order to register the daily physical activity, also the sleeping activity, the latency period (the period before sleeping), the awakening period § Sleep. Efficiency=Total. Sleep. Time/Total. Minutes. In. Bed § Input data – regards the daily physical activity; Output data – regards the quality of the sleep § Employ multiple deep-learning models & logistic regression for comparison • Multilayer Perceptron (MLP) • Convolutional Neural Networks (CNN) • Recurrent Neural Networks (RNN) • Long short-term memory (LSTM) RNN • Time-batched long short-term memory (TB-LSTM) RNN => Linear regression performed worse than the deep learning models; the best performance - CNN

Specific Tools for Data Mining q Generic data mining tools § WEKA, Waikato Environment for Knowledge Analysis – Data Mining Software in Java http: //www. cs. waikato. ac. nz/ml/weka/ § IBM SPSS Software – data mining methods and statistics https: //www. ibm. com/analytics/us/en/technology/spss/ § SQL Server Analysis Services https: //technet. microsoft. com/en-us/library/ms 175609(v=sql. 90). aspx § Deep. Learning 4 j https: //deeplearning 4 j. org/ q Specific for PLM § APRISO Warehouse Management http: //community. clujit. ro/display/TEAM/Baxter+and+data+mining § STATISTICA Enterprise https: //statisticasoftware. wordpress. com/tag/manufacturing-execution-systems-mes/

Conclusions Ø Data mining finding patterns and subtle dependences within data • Descriptive and predictive data mining • Employs various methods – supervised/unsupervised classifiers & deep learning, association rule mining, data visualization methods, statistics [1] § Appropriate to be used in PLM context within all the phases – BOL, MOL, EOL [3] • Identify the best values of the relevant parameters and the relationships among them in order to achieve maximum performance => multi-objective optimization [6], [7], [8] • Perform automatic corrective and predictive maintenance [9] • Choose the most appropriate model => highest robustness and comprehensibility (interpretability)

References [1]R. Kohavi, “Data Mining and Visualization”, National Academy of Engineering (NAE) US Frontiers of Engineering, 2000 [2] M. Najafabadi, F. Villanustre, T. Khoshgoftaar, N. Seliya, “Deep learning applications and challenges in big data analytics”, Journal of Big Data, Vol. 2, No. 1, pp. 1 -21, 2015 [3] J. Raphael, E. Schneider, Simon P. and E. , I. Sklar, “Behaviour Mining for Collision Avoidance in Multi-robot Systems”, Extended Abstract, 2014 [4] R. Chuchra, R. Kaur, “Human Robotics Interaction with Data Mining Techniques”, International Journal of Emerging Technology and Advanced Engineering, Vol. 3, No. 2, February 2015 [5] J. Li, F. Tao, Y. Cheng, “Big Data in product lifecycle management”, International Journal of Advanced Manufacturing Technology, May 2015 [6] Amos H. C. Ng, , S. Bandaru, M. Frantzen, “Innovative Design and Analysis of Production Systems by Multi-objective Optimization and Data Mining”, 26 th CIRP Design Conference Proceedings, pp. 665 -671, Vol. 50, 2016 [7] C. Gröger, F. Niedermann, and B. Mitschang, „Data Mining – driven Manufacturing Process Optimization“, Proceedings of the World Congress on Engineering 2012 Vol III WCE 2012, July 4 - 6, 2012, London, U. K. [8] K. Sun, Y. Li, U. Roy, “A PLM-based data analytics approach for improving product development lead time in an engineer-toorder manufacturing firm“, Mathematical Modeling of Engineering Problems, Vol. 4, No. 2, pp. 69 -74, June 2017 [9] S. Ren, X. Zhao, “A predictive maintenance method for products based on big data analysis”, International Conference on Materials Engineering and Information Technology Applications (MEITA 2015), pp. 385 -390 , 2015 [10] H. Oliffa, Y Liua, “Towards Industry 4. 0 Utilizing Data-Mining Techniques: a Case Study on Quality Improvement”, The 50 th CIRP Conference on Manufacturing Systems, Proceedings, pp. 167 -172, 2017 [11] A. Sathyanarayana et al, “Sleep Quality Prediction From Wearable Data Using Deep Learning”, JMIR Mhealth Uhealth, Vol. 4, No. 4, 2016, online: https: //www. ncbi. nlm. nih. gov/pmc/articles/PMC 5116102/ [12] R. Fehrer, S. Feurriegel, “Improving Decision Analytics with Deep Learning: The Case of Financial Disclosures”, Preprint submitted to ar. Xiv, August, 2015