Execution Monitoring Notification Aychin Gasimov DOAG 2016 Enterprise

Execution Monitoring Notification Aychin Gasimov DOAG 2016

Enterprise Manager Cloud Control Pros Cons • Wide range built-in metrics • Reach functionality • Includes job system • Incident management and reporting • Not just monitoring but also management • Configuration Complexity • BUGs and week support

Database can send incidents and problems saved into ADR to Enterprise Manager Incident Rule Set Check Incident occures Apply threshold Store in ADR Generate event New Event Send to EM Diagnostic Incident

Generate Event Define metrics Process Event

Design flow Admin Groups Templates Event Rules Jobs Credentials

Admin Groups Templates Event Rules Credentials Jobs Define hierarchy and create Admin Groups § § Groups targets by their type and properties Targets become the member automatically Automates target management on subgroups level Each entity is separate admin group ADMGRP Lifecycle Status Target type Database Instance Cluster DB Production Mission Critical Development Host ASM Cluster ASM Staging Test

Admin Groups Templates Event Rules Credentials Jobs Create Monitoring Templates and Template Collections § § § One template per target type and lifecycle status Centralized metric management Starting from 12. 1. 0. 4 define Adaptive Thresholds Group Monitoring Templates to Template Collections Associate Template Collections with Admin Groups Template Collection for Prod ADMGRP Lifecycle Status Target type Mission Critical Staging Database Instance Cluster DB Development Host Production ASM Test Cluster ASM

Admin Groups Templates Event Rules Credentials Jobs Create Event Management Rulesets § Associate with Admin Groups § Define § Target availability § Metric alert § Metric Evaluation Error § Define rule for notifying about Problems (from ADR) § SMS notification for Mission Critical and Production targets § Create incidents for specific conditions New event Action Lifecycle Mission Critical Send SMS and email Performance Send email Create Incident System Response time Development Capacity Send email

Admin Groups Templates Event Rules Jobs Credentials Create Jobs Use Jobs to execute OS scripts Create Jobs in Job Library Define them as “OS Command” and “Command Type” as “Single Operation” Define E-mail notification § In Job definition (notify only owner) § In Incident Rules Associate with Admin Group § Submit from library to activate as required § Associate with Admin Groups § §

Admin Groups Templates Event Rules Credentials Jobs Create Job to sync central folder over enterprise § Maintain one place to manage all scripts § Synchronize central location to local hosts § Benefits: § Ease of maintenance § High availability § High redundancy § Apply the Job to the highest level of the Admin Groups § Execution logs can be synced back to central folder Management host /home/oracle/scripts /home/oracle/logs Management host /home/oracle/scripts Central location for scripts and losg /home/oracle/logs Management host rsync -a --delete user@host: SOURCE DEST /home/oracle/scripts /home/oracle/logs

Admin Groups Templates Event Rules Jobs Credentials Define credentials for the targets § § Create Global Named credentials for MC and Production databases Use same global credentials for Primary and Standby databases as default ones Define “Database Host Credentials” for Database Targets, they used by Jobs If some target types mostly use same passwords, then define default credentials for them

Adding new target to EM Cloud Control 1. Save target 2. Set credentials 3. Set Lifecycle Status Database Instance DI template Host template Cluster DB template Lifecycle Status = Production Contact = BOTH Template Collection Prod-Grp Prod-DI-Grp Incident Ruleset for Prod-Grp Database Instance Jobs on Prod -Grp

General recommendations § Use corrective actions, specify them in templates § To fix issues § To gather additional information § Wait some time after configuration to filter out useful metrics and adjust thresholds § Check already generated incidents and add them to your rules if required § Backup important components periodically and before major changes. Use emcli utility with next options: § export_admin_group § export_incident_rule_set § export_jobs § export_metric_extension § export_template § Backup OMS configuration (emctl exportconfig)

Availability problems § § § Monitoring should be uninterruptable Scripts should be able to run without interruption EM 13 c provides new Always On Monitoring (AOM) Configure Agent Out-of-Bound notification Job executions can be offloaded to local crontabs AOM Target Availability events AOM Scripts running over Job system Metric Alerts Custom Solution

Custom solution for Jobs availability § Custom script resync_em_to_cron. sh § Extracts data from EM repository § Converts it to crontab format § Appends to crontab § Create EM Job which calls the above script with “add” option § Create EM Job which calls the scripts with del option EM Jobs EM Repository DB … add %Target. Name% DEPLOY_JOBS_TO_CRONTAB UNDEPLOY_CRONTAB_ENTRIES … del resync_em_to_cron. sh hosta resync_em_to_cron. sh hostb hostc resync_em_to_cron. sh hostd

Questions https: //aychin. wordpress. com/downloads