CERN Openlab Summer Student Programme Building effective Database
CERN Openlab Summer Student Programme Building effective Database Backup & Recovery Monitoring using Elastic Stack Author : Yasmine NASRI Supervisor : Sebastien MASSON August 15 th, 2017
Outline 1. Introduction 2. Elastic stack 3. Implementation 4. Conclusion 6. Future work
1 INTRODUCTION
PROJECT CONTEXT Experiments Administrative Oracle Databases Accelerators Backup & recovery system 2
PROJECT CONTEXT Logs generation Logs Analysis 2 3 4 1 Backup & Recovery Operations Monitoring of Backup & Recovery activities 3
PROBLEM STATEMENT Knowledge loss Notifications via email Loss of time in the analysis Analytical difficulties of large volume of logs Monitoring difficulties 4
OBJECTIVES Analyze log errors and extract knowledge form it Improve proactiveness Provide a smart alert system Provide statistics of backup & recovery activities 5
2 THE ELASTIC STACK
THE STACK’S GOAL Search Store it Processes, transform and enrich it Any source, any format Analyse The Elastic Stack Visualize 7
COMPONENTS OF THE ELASTIC STACK 88
PROCESS OF LOG ANALYSIS WITH ELASTIC STACK 1 4 Collect Store, search & analyse 2 Beat input plugin & Visualize manage 3 Filter plugin Transform Output plugin 98
3 IMPLEMENTATION
![IMPACT OF USING ELASTIC STACK BEFORE AFTER $ grep ' 30[1234] ' /var/logs/apache 2/access.](http://slidetodoc.com/presentation_image/3b1fa02c3c457371c8bc9cac19250cce/image-13.jpg "IMPACT OF USING ELASTIC STACK BEFORE AFTER $ grep ' 30[1234] ' /var/logs/apache 2/access.")
IMPACT OF USING ELASTIC STACK BEFORE AFTER $ grep ' 30[1234] ' /var/logs/apache 2/access. log | grep -v baidu | grep -v Googlebot … http: //logs. cern. ch: 5601 HELP! 11 8
LOG STRUCTURED 12 8
VISUALISATION OF LOGS 13 8
4 CONCLUSION
CONCLUSION Collecting logs with beats Parsing logs with logstash Centralized logs into elasticsearch Visualize logs with Kibana 15 8
5 FUTURE WORK
FUTURE WORK Visualize the log analysis results with dashboards Analyse the logs history and identify some trends Predict the error logs with machine learning algorithms 17 8
THANK YOU FOR YOUR ATTENTION
QUESTIONS? CONTACT Yasmine NASRI Computer Science engineer yassimene. nasri@gmail. com yasmine-nasri yassimene
![REFERENCES [Gormley & Tong, 2015] Gormley, C. Tong, Z. (2015). Elasticsearch: The Definitive Guide.](http://slidetodoc.com/presentation_image/3b1fa02c3c457371c8bc9cac19250cce/image-22.jpg "REFERENCES [Gormley & Tong, 2015] Gormley, C. Tong, Z. (2015). Elasticsearch: The Definitive Guide.")
REFERENCES [Gormley & Tong, 2015] Gormley, C. Tong, Z. (2015). Elasticsearch: The Definitive Guide. [Sharma, 2016] Sharma, C. (2016). Beginning Elastic Stack. https: //www. elastic. co/ https: //www. oracle. com/ https: //oracle. developpez. com/guide/sauvegarde/rman/
- Slides: 22
