Oracle Fleet Management Patch and Upgrade Your Databases

Oracle Fleet Management: Patch and Upgrade Your Databases Gary Henderson Vaithianathan Soundararajan

Nationwide Insurance and Financial Services Company Contents Fortune 100 Company Founded in 1925 43 Billion in Revenues* 221 Billion in Assets* 33, 135 Employees* #53 on Fortune’s "Best Companies to Work For" #27 in Computer World’s "Best Places to Work in IT. " *Fortune 500 Nationwide Database Teams

AGENDA • Why Patch? • Our Configuration • Patching Alternatives • Our Constraints • Our Results • How We Did It • Our Timeline of Progress • Summary • Questions

SECURITY. . SECURITY Center for Internet Security (CIS) 1 Oracle Database Installation and Patching Requirements One of the best ways to ensure secure Oracle security is to implement Critical Patch Updates (CPUs) as they come out, along with any applicable OS patches that will not interfere with system operations. 1. 1 Ensure the Appropriate Version/Patches for Oracle Software Is Installed Profile Applicability: • Level 1 -. . . The Oracle installation version and patches should be the most recent that are compatible with the organization's operational needs. Rationale: Using the most recent Oracle database software, along with all applicable patches can help limit the possibilities for vulnerabilities in the software, the installation version and/or patches applied during setup should be established according to the needs of the organization. Ensure you are using a release that is covered by a level of support that includes the generation of Critical Patch Updates.

Oracle’s Annual Release Roadmap Doc ID 742060. 1

2014 Clustering Technologies vs DB Versions 11. 2. 0. 4 1% 11. 2. 0. 3 36% RAC Multi. Node 17% 12. 1. 0. 2 1% 9. 2. 0. 7 1% 10. 2. 0. 5 12% Non-Clustered 34% RAC 1 Node 17% Veritas 32% 11. 1. 0. 7 19% 11. 2. 0. 2 31%

2016 Clustering Technologies vs DB Versions Non-Clustered 14% RAC Multi. Node 16% 10. 2. 0. 5 1% 11. 1. 0. 7 6% 12. 1. 0. 2 51% Veritas 10% RAC 1 Node 60% 11. 2. 0. 4 39% 11. 2. 0. 2 11. 2. 0. 3 1% 2%

2018 Clustering Technologies vs DB Versions Non-Clustered 9% 18. 3. 0. 0 10. 2. 0. 5 1% 0% RAC Multi. Node 12% 11. 2. 0. 4 9% 12. 1. 0. 2 90% RAC 1 Node 79%

• Out of Place • Version 1: Switch • Version 2: Db software maintenanc e • Gold Image Homes / Standardiza tion • Minimal Outage window • Utilizes OEM’s deployment job system • Scalable • *Life. Cycle License Required Cloud DBaa. S • Out of Place • Gold Image Homes/ Standardiza tion • Minimal Outage window • *Life. Cycle License Required? OEM Fleet Maintenance • Out of Place • Unplug from old and plug into updated/pat ched • May require double the memory resources • *Multi. Tenant license required Rapid Home Provisioning • In place • Challenging error recovery • Longer Outage window • All databases in the Home have to be patched together • Rollback challenging • Requires less storage than out of place options • *no extra licensing Multi-tenancy Opatch Patching Alternatives • Cloud Provide r applies Patche s

Quarterly Patching Windows Switch/Update Using a team of three: DBA – Db patcher DBA – Clusterware patcher/Tester OEM - Administrator Each Quarter patch to previous quarter’s PSU Development patched during Wednesday window Production patched during IRW window

PATCHING RESULTS Tot Exec Time Window Time (Hrs) Windows Success % # Dbs 2016: Devl 31 100 693 98. 15 38. 35 2017: Devl 44 99 1080 160. 67 52. 77 2018: Devl 42 100 1575 183. 56 48. 14 2016: Production 12 98 340 44. 36 18. 34 2017: Production 12 100 706 79. 38 20. 85 2018: Production 10 99 1000 105. 35 18. 24 11

OEM Fleet Maintenance (EMCLI) Version 1: Switch • SWITCH_DATABASE • SWITCH_GI Version 2: Db_Software_Maintenance (Update) • create. Software. Image • subscribe. Target • check. Applicability • perform. Operation • DEPLOY_GI_SOFTWARE • UPDATE_GI • DEPLOY_DB_SOFTWARE • UPDATE_DB • DEPLOY_RAC_SOFTWARE • UPDATE_RACDB • DEPLOY_CDB • ATTACH_CDB • ROLLBACK_DB …RACDB. . GI • CLEANUP_SOFTWARE Fleet Reference: https: //bit. ly/2 S 4 FZsz

Software Standardization Advisor

DB Software Maintenance Create Gold Image/Version create. Software. Image Prior Quarter When the old home is no longer used. . Remove it. CLEANUP_SOFTWA RE Patch Database moving it from the old to the new patched home UPDATE_RACDB Associate Target to Corresponding Image subscribe. Target One time (two weeks prior) Deploy new Oracle Home DEPLOY_RAC_SOFTW ARE Two Weeks prior

Gold Image Version – Lineage/Swim Lanes 12. 1 DB Standard RAC 12. 1 DB One-Off RAC Ver 1 180417 12. 1 DB Standard Stand. Alone Ver 1 180417 Ver 2 180717 Ver 3 1810? ? 11. 2 DB Standard RAC Ver 1 180417 12. 1 DB One-Off Stand. Alone Ver 1 180417 Ver 2 180717 Ver 3 1810? ? 12. 1 Grid Standard RAC Ver 1 180417 11. 2 DB Standard Stand. Alone Ver 1 180417 12. 1 Restart Standard Stand. Ver 1 Alone 180417 Ver 2 180717 Ver 2 180717 Ver 3 1810? ? Ver 3 1810? ?

Clus_server 1 Clus_server 2

Our Automation Leveraging Db Software Maintenance

PROGRESS – CRAWL, WALK, RUN … FLY 2014 Gold Image Provision of Database Homes RAC Clusterware Upgrade from 11. 2 to 12. 1 (60) 2015 Some Database upgrades 11. 2. 0. 2/3 to 11. 2. 0. 4 2016 Q 1 & Q 2 Mandate ALL DB homes OEM provisioned Mandate ALL RAC Clusters built with OEM Continue upgrades 11. 2. 0. 4 & 12. 1. 0. 2 Q 3 & Q 4 Patch 1033 databases using Switch Database 2017 2018 1786 databases patched using Switch Database 2575 databases patched (so far) All databases patched in 2 nd quarter 3 rd Quarter moved from Switch to DB_Software_Maintenance 18

• SUMMARY: • RECOGNITION: • Missing Nationwide team member John Norman • Oracle OEM Product Team: Harini Srinivasan Bharat Paliwal Martin Pena Saurabh B Jain • Oracle OEM Dev Team: Harmeet Kaur Paras Narang Karthika Thirumalasamy • • Oracle Senior Enterprise Account Executive. Stephan Saade • Oracle Key Account Director for Nationwide Johnston • OTHER SESSIONS: • Hands On Lab (HOL 6350) Next-Generation Database Patching - Wedneday 11: 15 • OEM kiosks at the Demo Grounds (MGMT-WU 2) • Oracle Fleet Reference Manual: https: //bit. ly/2 S 4 FZsz QUESTIONS ? ? ? 19