Quarterly Enterprise Collaboration Meeting August 6 2015 11

Quarterly Enterprise Collaboration Meeting August 6, 2015 11: 00 a-12: 00 p

Agenda 1. IS Ops Service Overview Document KB 0020368 (5 minutes) § Presented by Wes Rodriguez 2. Partners IS Command Center Overview and Updates (15 minutes) § Presented by Michael Cole 3. Partners IS Warehouse Operations Overview (15 minutes) § Presented by Christopher Tentindo 4. Partners e. Care Infrastructure Group 1 Readiness Overview & Group 2 Planning (15 minutes) § Presented by John Grieco 5. Q/A and Feedback for IS Ops Leadership (10 minutes) § All 2 Information Systems

IS Ops Service Overview Document Wes Rodriguez Service Catalog Manager 3

Service Overview Document § The Service Overview Document was created to exhibit the Services currently offered by IS Ops § The majority of these Services are Technical Services § The document receives at least one revision per quarter, and it lives within the Service. Now Knowledge Base 4 Information Systems

History and Where We Are § First introduced in January, 2015 § Basic descriptions of IS Ops Services § No categorization or meaningful structure to Services § Last updated in July, 2015 § Document has doubled in size § Descriptions are much more robust, majority of Services include delivery and quality expectation, and price/cost/charge back information § Services have been meaningfully categorized 5 Information Systems

What’s Next § Inclusion of more IS Ops Services § Continued fine-tuning of Service Categories § Distinction between IT Business Services and Technical Services § Categories and Services will be built in Service. Now and made available to our customers as a true Service Catalog § Eventual retirement of document 6 Information Systems

Thank You Wes Rodriguez Service Catalog Manager wrodriguez 4@partners. org Service. Now: service catalog mgmt. – phs (please open a request) 7 Information Systems

Command Center Overview Michael Cole Corporate Manager, Network Engineering

Command Center Objective Key Command Center Objective The overall objective of the Command Center is to have a proactive and Collaborative functional monitoring and support center with SME capabilities. A consolidated group for which Event and Incident Management can reduce MTTD, MTTR, impact to the Business and Resolve 90% of tickets within CC. 9 Information Systems

Command Center Scope Hours of Operation: Monday – Friday 8: 30 AM to 5 PM 5: 01 PM to 8: 29 AM and weekends/holidays – Support groups follow their escalation procedure to On-Call Scope: § Monitor the IS supporting infrastructure and a few applications (Exchange, AD, DNS/DHCP & Vo. IP) § § Perform ITIL Event and Incident Management § § § Neteng, Net. Sec, Telecom, Server, SQL and Storage infrastructure Resolution of all Incidents (SD and Monitoring Alerts) for Net. Eng, Net. Sec, Server and Telecom groups Piloting Problem Management Support Knowledge Management Support Change Management Provide Continual Service Improvement (CSI) Not in Scope: § § Not a Service Desk Ownership of Major Incident Process Service Requests Applications not being monitored by Command Center – i. e. Epic, LMR Resources: § § § 10 Command Center Manager 4 permanent and 2 rotational engineers from Net. Eng, Server, Open Systems, Net. Sec, Telecom groups Monitoring Tools team Information Systems

Command Center Overview § People § 4 permanent resources from Net. Eng and Server teams – Resources availability § § Server – 6: 30 AM to 6: 00 PM and Net. Eng – 7 AM to 5: 00 PM Processes § § P 1 and PCC SD tickets paged to CC – Server only – Net. Eng and Telecom coming soon Critical and Proactive Alerts are prioritized and addressed immediately Implemented Command Center Priorities and SLOs – baseline MTTD & MTTR Number of tickets created/worked increased by 161% (Average) (FY 2015 Q 1 to 3) § Growth as a result of increased number of devices monitored and Incidents from SD § Freeing up support teams to focus on projects and high-value tasks § § CC Escalation to outside support groups decreased by approx. 50% Increased cross communication with external groups – onboarding process Onboarding in progress for SAN, PEC, VBLOCK and Nagios Tools § Monitored devices increased in SMARTS, SRM and SCOM § § § 11 Information Systems Servers, SAN, Routers and Switches – 6, 600 Phones – 5, 000 40, 348 8, 850 Monitoring enhanced for § SAN, PEC, VBLOCK, Telecom, Net. Eng and Wireless § Onboarding in progress for SAN, PEC, VBLOCK and Nagios Command Center and outside support groups are leveraging consolidated alerts consoles Number of non-actionable alerts removed as result of CC monitoring and feedback – WIP

Total Tickets Worked Number of tickets created/worked increased – 161% (Average) 12 Information Systems

Total tickets – 2014 and 2015 FY 2014/15 – Q 1 & 2 Total Tickets ** Total Tickets by Function ** 13 Information Systems

Q 1 & 2 – Monthly Tickets FY 2014/15 – Q 1 & 2 Monthly Total Tickets ** Total Tickets by Command Center and Escalated by Service Desk ** 14 Information Systems

Command Center Escalations FY 2015 – Oct to June Escalations ** Objective is to resolve 90% of the tickets within Command Center ** - Typical groups tickets escalated to – Citrix and Exchange, …. - Warm Transfer process is followed for escalated Service Impacting Incidents 15 Information Systems

Command Center Metrics – Past Method FY 2015 – Q 1 & 2 Monthly MTTD & MTTR ** Objective is to meet MTTD & MTTR SLOs ** - The MTTD and MTTR are during Command Center operational hours – 8: 30 AM to 5: 00 PM - MTTD and MTTR are related to the total number of tickets worked by Command Center. Currently, we see an increase in Total Tickets, MTTD and MTTR from efforts to clean up older/lower priority alerts. 16 Information Systems

Command Center SLO Metrics Command Center – MTTD & MTTR by Priority and SLOs ** Objective is to meet MTTD & MTTR SLOs ** 17 Information Systems Breaches in SLAs will be identified by a red background.

Sample Command Center Metrics Weekly Tickets – MTTD & MTTR by Priority ** Objective is to meet MTTD & MTTR SLOs** MTTD 18 Breaches in SLOs will be identified by a red background. Information Systems MTTR

Challenges § Command Center § SD quality of tickets & misclassifications – Lack of actionable information – Increases MTTR § Teams are using varied support models and processes – Impacting escalations, communication, access, etc. § Maintenance mode not being used – wasting resources time § Device status validations after ISM – needs improvement § Telecom skill-sets and monitoring not consistent to CC requirements § Removing non-actionable alerts are taking longer than expected § Engaging Command Center and Monitoring Tools on new initiatives and project initiation § Monitoring and CC onboarding are impacted § Onboarding other technologies taking longer – e. g. SAN, VBLOCK § Challenging to identify, configure, and work (KB) actionable alerts § No 24 x 7 coverage – results in raising MTTD and MTTR § Monitoring Tools § § 19 Information Systems Improvements/Enhancements stalled or taking longer as a result of limited resources Number of Alerts impacting SMARTS availability Onboarding technologies/tools taking longer – PEC, SAN, VBLOCK and Nagios Monitoring Tools team not engaged during project initiation

Functional View – Current State Information Systems

Functional View – Target State Information Systems

Command Center Future Goals § Command Center § § Acquire applications to infrastructure mapping knowledge – CMDB Standardized Onboarding process for CC and Tools § Working on onboarding PEC infrastructure, SAN, VBLOCK and Nagios environments § § § Working with SD to improve ticket quality to reduce MTTR Established steering committee for Monitoring Tools and Command Center Enhance Proactive Monitoring § Alert cleanup effort enabling time for proactive monitoring § § § Enhance KB – to improve 90% resolution rate within CC Develop interface agreement with the DR team Establish an interface with MSSP and SOC team/processes Work with facilities and other team towards Assembly Row Command Center Work with other support groups on communications, Roles and Responsibilities, and a common support model Monitoring Tools § Onboard the following technologies/environments § PEC infrastructure, SAN, VBLOCK and Nagios § Implement SQL and Citrix modules in Extra. Hop appliance for historical and realtime monitoring of PEC infrastructure environment in Needham (ONLY) § Alert Cleanup – remove/filter non-actionable alerts from SMARTS – WIP § Improve and enhance monitoring for § Wireless and Telecom infrastructure 22 Information Systems

Q&A Information Systems

Monitoring Tools Functions • Monitors – Network, Severs, VMs, Open Systems, Firewalls, etc. • Topology Discovery and Correlation • Device up/down • CPU and Memory • Disk Space and File Space • Environmental – Fan, Power Supply, Temp, etc. • Network Interface • Routing – OSPF, BGP and MPLS • Processes – only for Vo. IP • Monitors – Windows Severs • Device up/down • CPU • Memory • Disk Space • File Space • Network Interface • Processes or services running on a device SCOM SMARTS • Notifications/paging • Email • Paging - Cell phone or PPD • SMS • Built in escalation capabilities with 2 way message stopping • Built in scheduler for staff and/or time of day criteria Attention 24 Information Systems • Monitors – Network, Severs, VMs, Open Systems, Firewalls, Storage, etc. • Service Dashboards • Historical Performance graphs • Capacity Management reports/graphs • Scheduled/Ad-hoc Reports • Alerting to SMARTS via Traps SRM

Interactions between Event & Incident Information Systems

Partners IS Warehouse Overview Christopher Tentindo Information Systems Warehouse Manager “Do not tell me what you can do, show me what you can do”

IS Warehouse– Services § § § § 27 Customer Service and Guidance-Enterprise Wide Order Placement Review-Enterprise Wide Work Order Generation-Print Queues, Wiring, Moves, Installs Receiving/Distribution of All New Equipment/Repairs-Enterprise Wide Recovery of All Old Equipment/Repairs-Enterprise Wide Initiating/Completing Imaging Process Disposal/Wiping of Data-Enterprise Wide Hardware Repair of Workstations/Laptops-Enterprise Wide Processing Exchanges/Returns with Purchasing/Vendors Asset Tracking and Partners Inventory Advocate on behalf of our Customers with all Vendors Project Management of Partners Inventory for IS/MGH Upgrade All Other Duties As Assigned Information Systems

IS Warehouse/IS Customer Service End User Guidance Order Placement Review IS/MGH Upgrade Project Work Order Generation Print Queues Wiring Request Port Activations Moves Installs 28 Information Systems

IS Warehouse Imaging Workstations Laptops Repairs Collaborate with CID on Imaging Process Maximum imaging capacity 512 Desktops or 128 Laptops Or a combination there of daily Enterprise Wide 29 Information Systems

Partners IS Warehouse Receiving/Shipping • Receive Product Into Partners/DFCI Systems • Assign Specific devices to TASK for Imaging • Monitor Time Tolerances for Order/Product Turnaround Times • Escalate Order Processing as Needed for Time Sensitive Shipments/Projects • Ship and Track all Product in Service Now • Collaborate with Partners Transportation on Logistical Needs 30 Information Systems

Partners IS Warehouse Receiving/Shipping • Interact with Partners Purchasing as an Advocate on Behalf of our Customers • Periodically Review and Identify Cost Saving Methods • Maintain Inventory/Asset Tracking of Customer Owned/Project Owned Items • Coordinate with Multiple Team Leads/Project Managers, Technicians and our Most Valued Customers Throughout the Enterprise • Error Rate of Less than. 01% 31 Information Systems

Partners IS Warehouse Operations/Repairs • Track Incoming/Outgoing Repairs • Advocate on Behalf of the Customers Best Interest with Ergonomics and Hewlett Packard • Collaborate with Partners Purchasing and Customer to Generate Required Purchase Order for Repairs Not Covered by Warranty • Escalate Repairs as Needed • Record Metrics and Make Process Changes to Drive Efficiency 32 Information Systems

Partners IS Warehouse/Partners Transportation • Distribution of All New Equipment/Repairs-Enterprise Wide • Recovery of All Old Equipment/Repairs-Enterprise Wide • We Deliver to an average of 15 -19 locations daily. • 95% Institution/Building moves occur during deliveries/pick ups • 90% of our volume is handled by one driver/one truck with the balance handled by Partners vans 33 Information Systems

Partners IS Warehouse Operations Evolution 2000 CNY 149 2014 121 Innerbelt Rd § § § § § § § § Employees: 6 Institutions Serviced: 1 Delivery Locations: 5 Units Imaged Daily: 10 Image Process: 5 CD Install Work Orders: 3, 639 Units Processed Yearly: 4, 600 Receiving/Shipping Methods: Paper Shipments Per Year: 1, 658 Shrink: $120, 000 year Turnaround for New Device: 3 months Warranty Repairs: 1, 725 Vendor RMA: 109 Disposal: 268 Employees: 7 Institutions Serviced: Enterprise Wide Delivery Locations: 359 Units Imaged Daily: 90 -120 Image Process: 32 g USB Key Drive Work Orders: 12, 949 Units Processed: 101, 798 Receiving/Shipping Methods: Electronic Shipments Per Year: 8, 275 Shrink: $Zero$ Turnaround for New Device: 10 -14 days Warranty Repairs: 1, 225 Vendor RMA: 179 Disposal: 11, 687 Units process through 2014: 930, 202 Work Orders includes: Device, Network Printer, Data port, Jack Activations January-December 2014 34 Information Systems

Partners IS Warehouse/Hardware Disposal • Recovery of Equipment • Retain Usable Parts to Keep Non-Warranty Repair Cost Down • Overwritten Twice, Hammered to Ensure No Data is Recoverable Between Partners and Disposal Vendor Before Hard Drives are Shredded • Scanning/Recording Disposed Assets • Leverage Recyclables for Profit Sharing to Keep Disposal Cost Neutral Enterprise Wide 35 Information Systems The Infamous Sledge Hammer Like the Legend of Big Foot Always Heard of, But Never Seen

Partners IS Warehouse Operations 2000 -2014 Partners IS Warehouse Yearly Volume 120000 101798 96998 100000 75856 80000 67051 60000 53223 83185 80379 78154 71550 57745 47463 38095 40000 39492 25593 20000 13580 0 YTD-01 YTD-02 YTD-03 YTD-04 YTD-05 YTD-06 YTD-07 YTD-08 YTD-09 YTD-10 YTD-11 YTD-12 YTD-13 YTD-14 January 2000 through October 2014 36 Information Systems

Partners IS Warehouse-Always Looking to Drive Improvements Most Recent Past Projects Future Projects § § § Online WO Form Redesign § Service Ticket Auto Generate a Service Now Task § Auto Generate Shipping Documents by Leveraging Service Now § § IKON Printer Process Storage Space for Restore Disk “Go To My PC” Renewal Process Pull Up User Info on WO’s by User ID. Network Printer WO to Include User Information MDOP License Purchase Process Always looking to make improvements that will save money, time and improve process. 37 Information Systems

IS Warehouse Collaboration • Customers • Multiple Vendors • Partners Purchasing • Partners Transportation • Network Engineering • Service Desk • Client Infrastructure Design • Server Group • Support Services • Project Managers • Team Leads • Telecom • Technicians • Research • Information Security Enterprise Wide 38 Information Systems We must work smart, be efficient and maneuver wisely Others rely upon us in more ways than you can imagine.

Pe. C Infrastructure Group 1 Readiness & Group 2 Planning John Grieco Associate Director, Information Systems Technical Services and Operations

Agenda § Pe. C Infrastructure Fact Sheet / Topology § Printers and Workstations § Command Center Facts from DFCI and OCC § Go Live Information § Ticket Stats from the first month § Group 2 Planning 40 Information Systems

Pe. C Infrastructure Design Build Committee § Meet every Thursday from 9 am -10 am at OCC and Web Ex § Multi-Disciplinary Infrastructure Teams and Business Partners Attend § Guidelines of the Committee § Pe. C Infrastructure Build Timeline 41 Information Systems

Pe. C Infrastructure Build Timeline 42 Information Systems

Pe. C Infrastructure Fact Sheet 43 Information Systems

Partners e. Care – High level Platform Topology 44 Information Systems

Pe. C Core Infrastructure – Cache Backend (Active/Standby) 45 Information Systems

Pe. C Core Infrastructure - Citrix Xenapp Infrastructure (Active/Active) Citrix is currently designed as active/active (Currently runs 98/2 on the Net. Scalers with 98% out of NDC) Ø Architectural consideration: Configure Citrix Xenapp to be 50/50 on the Net. Scalers 46 Information Systems

Printers and Workstations § Collaboration with site IS resources § Technical Dress Rehearsal (TDR) lasted 8 weeks § 23, 000 workstations and 8000 printers were inventoried at hundreds of locations § 200 hours of team meetings were held during TDR 47 Information Systems

Pe. C Core Infrastructure – Epic Print Services (Active/Passive for NWH) § Architectural consideration: Configure to be 50/50 on the Net. Scalers 48 Information Systems

Pe. C Core Infrastructure – Epic Print Services (Active/Active for MGH) 49 Information Systems

Pe. C Core Infrastructure – LRS Printing (Active/Active) 50 Information Systems

Command Center: DFCI Infrastructure to support a peak volume of 415 users was prepared, including: § 3 temporary Cisco switches deployed to support Command Center area IP phones and computers § 10 Access Points added to support the high volume of users in conference rooms § 380 Vo. IP phones, 180 headsets, and 12 conference phones installed 51 Information Systems

Command Center: OCC 52 § 10 resources were on site, ready to respond, during cutover and go live activities § 7 vendors were on high alert standby (Citrix, HP, EMC, Intersystems, VMWare, IBM, and Epic) § Dedicated Pe. C consoles monitored 1300 infrastructure components such as: § Storage § Network § Servers (Windows and Open Systems) § Citrix § Custom SRM reports were configured for historical performance and capacity management for Pe. C infrastructure Information Systems

Command Center Video Wall Views Dedicated Pe. C console for monitoring the infrastructure components Extra. Hop Citrix Dashboard 53 Information Systems Net. Eng and Telecom console view. Shows Events and Incidents being worked on by the teams Server Console view. Shows Events and Incidents being worked by the team System. Pulse Console for Ecare monitoring Storage console view. Cisco CTC Console for monitoring fiber between major data centers

Go Live! 54 § 1800 total resources were in the Pe. C Staffing Plan, with 415 at peak, during the first 24 hours after go live § 38 were PHS-IS resources, with 11 present during the cutover and go live event § Activated 2800 Express Mode 2, and 150 Express Mode 1 machines § Activated 200 Express Mode 4 machines in support of fetal monitoring and status boards § Activated 70 full screen status boards § Moved 2000+ calls per day from legacy BWH/BWF Room Clean/Transport to Epic IVR § Added 120 Dragon users § Moved 5000 faxes a day from legacy fax server to Right. Fax § Peak user counts are now over 10, 000 Information Systems

Ticket Stats for All Go Lives • Over 31, 000 tickets since the Group 1 go live. • Peak day is consistently Monday after Go Live. • Difficult to trend on infrastructure because tickets moved between groups. Tickets Opened Per Day for All Go Lives 3000 2500 2000 1500 1000 500 0 MGH 55 Information Systems NWH NWAS Group 1

Group 2 Planning § Apply Lesson’s Learned § Disaster Recovery Failover from Needham to Marlborough Data Center § Command Center Application Monitoring § Provision Complementary Applications for Group 2 § CapacityPerformance Planning for Group 2 Connections 56 Information Systems