Best Practices at ATT Oracle Real Application Clusters

Best Practices at AT&T Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite Bryon Rickey Ravi Kayarthodi September 20 th, 2010 © 2010 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.

Agenda v Welcome and Introductions v Supply Chain Management (SCM) environment Ø Overview/highlights Ø Architecture v Partnership with Oracle to transform our Business Ø Key project highlights Ø Future initiatives Ø Solution Support Center (SSC) and on-site support v Q&A

About AT&T v AT&T is one of the world’s largest Telecommunications companies v AT&T Global Headquarters is based in Dallas, TX v AT&T has approximately 290, 000 employees v AT&T has more than 90 million wireless customers v AT&T has more than 16 million high speed internet subscribers v AT&T has more than 2. 5 million uverse subscribers v Exclusive U. S. provider of i. Phone 3 G / 3 GS / 4

AT&T SCM: Architecture highlights v Oracle E-Business Suite Centric v Scale-up with cluster database on very large infrastructure v High availability with Oracle Data Guard, Oracle Golden. Gate, and site level redundancy v Central inventory/financial management; utilizes product hub, planning/forecast with ASCP/Demantra v Order orchestration and brokering with in-house developed Oracle based application system v Internet deployment for interaction with external suppliers v Management reporting/profitability analysis; procurement and spend analytics v Industry leading warehouse/distribution management systems

AT&T SCM Platform: Financial Integration v Both Network and Retail SCM - financial counterparts are integrated in the same platform: Shared AP, AR, PA, FA, and GL v Supply Chain transactions leverage same referential data and accounting rules v Financial integration of supply chain transactions are seamless and are recorded real-time in the general ledger v No need for reconciliation between sub ledgers v Ability to forecast financial implications based on SCM forecasts v Online financial budgeting validation on purchasing and requisitions v Analytical GL Cube and Inventory Cube

Results v AT&T has saved more than $500 million and has improved its purchasing power by combining accounts with its vendors. v Estimated to save $1 B in inventory costs, and $300 M per year in reduced operating costs. v Warehouses have been able to keep inventories low without missing SLA, resulting reduced overall inventory costs. v Ability to identify sold merchandise, resulting in fewer fraudulent returns

Industry Recognition v 2010 IDG’s CIO Award v Recognizes organizations around the world that exemplify the highest level of operational and strategic excellence in information technology

Systems Architecture

AT&T SCM: E-Business Suite Centric

AT&T SCM: Key Systems Architecture ED I I ED e. Auction ODP OM ASCP Inventory Planning RNF Purchasing PO GL AP PA FA Financials Oracle E-Business Suite 11 i Receipts Shipments Warehouse Mgmt Distribution Center Receipts Moves Audits Repairs Inventory Tracking Corporate Asset Tracking

Oracle 11 i E-Business Environment Forward Orders Store Replenishment Planning Dealers/ i. Store National Retailers Purchasing Direct Fulfilment DSL and u. Verse HSIA Order Management Inventory Receivables Upfront EPL Financials Oracle E-Business Suite 11 i Apps Store Reverse XBM/Insurance Dealer/NR Service Depots RU 30 Sales Opus Returns Company Owned Retail 416 CPU Cores & 1. 7 TB of Memory in DB Cluster 14 TB of storage for transaction DB 3 TB of storage for Archived DB

AT&T SCM Platform: Oracle 10 g RAC Infrastructure: VCS+ CRS

AT&T SCM Platform: Transaction Volume v Approximately 3. 35 million logons to the production areas per day v 2. 06 billion SQL executions per day v Approximately 136. 5 Billion logical reads and 41 TB of physical reads performed per day v Approximately 7 TB of writes performed per day v Approximately 1. 05 billion user calls and 52 million transactions completed per day v 200, 000 concurrent request per day

AT&T SCM Platform: Operational architecture v Decoupled month-end book closing preparation Ø Statistics gathered on a clone v Separate reporting environment – refreshed nightly; Inventory and GL Cubes for reporting v Revenue share and allocation on a separate system v Direct supplier interaction with Internet-deployed application modules v Utilizes market making mechanisms (sourcing)

AT&T SCM: Oracle 11 i Dev/Test Configurations Bothell, WA HDS – USP-V Data Guard/ Custom Shipping Net. App GOLD L-TRA Data Guard Shadow Image HDS – USP-V PROD Alpharetta, GA Data Guard R-TRA Shadow Image Data Guard/ Custom Shipping Net. App GOLD MET 2 Net. App Master Flex Clone Snap. Mirror FTP Shadow Image Flex Clone DEV QA DEV RPT QA DEV QA MET 1 FTP QA Flex Clone DEV DEV v Storage Efficiency with Net. App de-duplication Tools v Performance test environment on tier-1 storage Net. App Master

AT&T SCM: Oracle Automatic Storage Management v Local and Remote DR databases and also Reporting databases have been migrated to ASM storage to meet storage and capacity needs v Avoided SFRAC costs v Performance v SCMP DR/BC database converted to RAC database v Saved SFRAC license cost on E 25 K frames v Comply with AT&T standard for cluster database (ASM is required) v Expected to save cost in the future when we upgrade other Corporate Systems’ databases

AT&T SCM: Archive and Purge Archive Stats v 16 custom and core modules including (GL, INV, OM, AR, WSH, PO, FA) v Executed weekly, monthly, quarterly, and yearly v History instance v Informatica Data Archive application used v Schedule is monitored real-time to ensure adherence Purge Stats v 70 jobs via Oracle concurrent programs v Executed daily and weekly v Schedule is monitored real-time to ensure adherence

AT&T Supply Chain Geographic Distribution – DB Flow to Test and Dev Primary Site XX XX Test Site 1 Development Site 2 XX XX Test Site 2 XX XX DR Site XX Prod Dev/Test XX XX XX Development Site 1 XX XX

AT&T SCM: Spend Analytics v Oracle Business Intelligence Enterprise Edition (Oracle OBIEE) Powered Top 5 Categories v Sourcing from Oracle E-Business Suite and Legacy Applications Spend $0 $2 $4 $6 Millions $8 Top 5 Suppliers Jan Feb Mar Apr May Jun $0 Line Of Business Jan Feb Mar Apr May Jun $5 Contract Leakage $10 Millions Off Contract Spend Payables Leakage

AT&T SCM: Database Maintenance/Performance Management v Monthly month-end close of books stats validation – database statistics preparation on production clone v Table Re-organization – routine and ad-hoc v Periodic index rebuild v Daily performance call v Daily/weekly/monthly/yearly purge/archive schedules – vigilance in adhering to them v Extensive performance monitoring on all levels of the system

Partnering with Oracle Advanced Customer Service Oracle Solution Service Center (SSC)

SSC: Project Spotlight: Patch Recommendations Project Activities o o Identification of available latest required patches for AT&T environment Participation in meetings to review recommendations Description An initiative to incorporate E-Business Suite recommended patches, such as, Rollup (RUP) in AT&T’s release schedules for 2010 Value Addition v SSC and AT&T worked together to create a new process to efficiently identify required patches with the goal to minimize the impact of rolling them out in our production environment and to current/future projects: v Quarterly document for AT&T environments with the latest recommended patches. v Participation on AT&T IT meetings to clarify recommendations and questions v Provide detailed patch analysis on a ad-hoc basis v Successful roll out of AP. O patchset along with other Financials patches in February 2010: stabilized production environment and brought environment to the latest code base. v Successful completion of the ASCP Rup 37, Collections Rollup (RUP) 32 and Application Technology Group (ATG) RUP-7 for the April 2010 production rollout. 23

SSC: Project Spotlight: Performance Improvement Project Activities o o o Understand AT&T batch window requirements and transaction volumes Identification and root cause analysis of problem areas Implementation of resolutions Description An initiative to address various performance issues related to Month. End-Close processes and other batch processes after Oracle 10 g database upgrade. Value Addition v Active participation on Conference bridge calls and OWC sessions. v Engage Oracle development groups (Financials and Apps Performance) to provide SQL tuning improvements. v Improve month end close activities like AR close, depreciation and asset retirement, optimizer, Invoice Validation processes. v Maximize performance during month end, preventing delay; AT&T was able to successfully meet the SLAs for the Month End Close Processes month after month. v Provide recommendations for major upgrades (DB, Applications and Tools)

SSC: Critical SR Spotlight: Other Key areas of focus Project Activities o o o Description Workflow background process WF_DEFERRED queue growing exponentially; AT&T engages SSC to evaluate and address the issue. upgrade. RAC – CRS Configuration Collect diagnostic and configuration data Troubleshoot frequent RAC node reboot/performance issue post upgrade Discover unforeseen problems Validate AT&T configuration with RAC Best Practices Device Life Cycle Database Upgrade Provided upgrade expertise and support (pre and post deployment for 11 g R 2 upgrade. Value Addition v v v Active participation on conference bridge calls and OWC sessions v Work with at&t and Oracle development organization to provide scripts to clear workflow DEFERRED queue v v v Monitor the implementation of recommended suggestions Review current setups and schedules of workflow Background Process Engage on Severity-1 SR and work collaboratively with Oracle development to provide workaround (eg. skip script) Engage Oracle Development and RAC COE team Engage on Architectural design discussions Stand-by support for DB/RAC upgrade deployment activities Ensure successful implementation of RAC upgrade Rollout

SSC: AT&T Targeted Services: Operations The Services Yield to the Customer targeted for the next 6 -month planning period is comprised of key Support activities, aligned with key customer objectives/activities towards achievement of AT&T’s Account Goals Supply Chain Transformation (Streamline Operations for newly deployed modules) v Work with ACS sales to determine appropriate support for upcoming SC rollout. Determine/secure funding for new modules (Demantra and Transportation Management Suite) v Develop, Enhance and Secure development sponsors v Participate in scheduled project “Deep Dive” calls with PMO and project team v Site visit to Memphis to understand warehouse and logistic operations – understand business functions. v Supply Chain Business – develop key relationships with Retail and Network business users 11 i Financials and Supply Chain Production Supportability (System) v Next generation i. Phone launch and 2010 Holiday Retail Season readiness v “Top Ten” Prioritization

SSC: AT&T Targeted Services: Transportation Manager One. ATT- New Horizon to transform Network and Asset Management: Logistics TMS Key Secondary Activities v Participate in scheduled project “Deep Dive” calls with PMO and project team v Work with PMO/SSC to determine appropriate support model for New Horizon’s v Work closely with OCS /PMO on project milestones – develop relationships v Continue to maintain Dev OTM Network v Supply Chain Business – develop key relationships v Site visit to Warehouses v SSC walk-thru’s Key Deliverables v Enhance Supply Chain SSC presence and understand usage and architecture for OTM v Site visit to understand implementation, document in configuration guide. v Performance testing as requested. v Amend SSC contract to include OTM products

SSC: Oracle E-Business Suite Release 12/Database 11 g R 2 Upgrade v Upgrade existing Oracle 11 i. 10 SCM and Planning environments to EBusiness Suite R 12. This will enhances supportability for our complex environment and provide additional needed functionalities v Three stage upgrade (2 – 3 year journey) § Supply Chain planning to release 12 § RDBMS for 10 g to 11 G § Supply chain platform to release 12 Objective: v System Availability, Supportability, and Software Currency Key Secondary Activities v SSC “Top Ten” Prioritization: Service Request review meetings with Supply Chain, Financials, and DBA support teams v Patching/Release Management strategy for Release 12 interoperability; v Leverage Tools and Templates v Daily SSC Performance Calls Key Deliverables v Proactive Diagnostics (Configuration/Performance) v Contracted deliverables v Semi-Annual Account Review

Solution Support Center Service Highlights v Focused Engineering Team of Oracle Subject Matter Experts v Fast track solutions based on intimate Knowledge of customer Environment and business needs v Proactive Software Advice and Knowledge Transfers v Performance and Patch Assessments v Single point of contact for all technical issues - Technical Lead v Single point of contact for all escalations – SDM v 24/7 coverage Value v Mitigate Upgrade Risks substantially v All project milestones are met v Ensured optimal usage of Oracle features and functionality v Reduced overall Maintenance/upgrade/ training costs

Solution Support Center Key Delivered Features ü Focused Engineering Team ü Single Point Accountability – SDM ü Exclusive 1 800 number 24 x 7 x 365 ü Review of Critical Issues as Needed ü Patch Analysis ü Stand-by Support for Critical Go-lives ü Proactive Alerts / Patch Notifications ü Project Planning / Migration Assistance ü Periodic Review Sessions ü Applied Research

Questions

For More Information On this presentation, e-mail Bryon Rickey (BR 3681@att. com) or Ravi Kayarthodi (RK 8409@att. com). For general information on Advanced Customer Services, go to oracle. com/acs or e-mail acsdirect_us@oracle. com