Challenges solutions and lessons learnt in 7 years

Challenges, solutions and lessons learnt in 7 years of Service Management at CERN David Martin Clavo CERN, IT Department, Computing Facilities Group, Service Management section 9 October 2018 HEPi. X Autumn/Fall 2018 Workshop Barcelona, Spain ©CERN CC-BY-SA 4. 0

Agenda Introduction Bringing services on board User experience Tool configuration Conclusion and Future 9 October 2018 ©CERN CC-BY-SA 4. 0 2

1. Introduction 100000 80000 60000 40000 20000 0 Q 1 2011 Q 2 2011 Q 3 2011 Q 4 2011 Q 1 2012 Q 2 2012 Q 3 2012 Q 4 2012 Q 1 2013 Q 2 2013 Q 3 2013 Q 4 2013 Q 1 2014 Q 2 2014 Q 3 2014 Q 4 2014 Q 1 2015 Q 2 2015 Q 3 2015 Q 4 2015 Q 1 2016 Q 2 2016 Q 3 2016 Q 4 2016 Q 1 2017 Q 2 2017 Q 3 2017 Q 4 2017 Q 1 2018 Q 2 2018 Q 3 2018 LHC Operation phase: more users, less staff Service Management growth Human Incidents Computing Monitoring Incidents 2000 1600 1200 800 400 0 20 1 1 20 12 20 13 20 14 20 15 20 16 20 17 20 18 25000 20000 15000 10000 5000 0 Requests Unique Callers 9 October 2018 ©CERN CC-BY-SA 4. 0 Supporters 3

2. Bringing services on board Service Catalogue • • • Two dimensional catalogue splits the what from the how Original challenge: convince service owners of value Catalogue size per year Solution: • Initial big bang + review • Now: step-by-step, handholding • Continuous improvement and maintenance of the catalogue are critical 1000 800 600 400 200 0 2010 IT 9 October 2018 EP 2011 2012 RCS-SIS SMB ©CERN CC-BY-SA 4. 0 2013 FAP IPT 2014 HR HSE 2015 BE EN 2016 TE IR 2017 DG PF 4 2018 TH

2. Bringing services on board Single process set • A single and simple set of processes enables scalability • Services typically added in ½ day • Opposite: “island” organization • Increased overhead • Different workflows • Ticket conversions IT Service Management HR Facilities 9 October 2018 Finance ©CERN CC-BY-SA 4. 0 5

2. Bringing services on board Contract management Challenge: measuring service quality, compliance and performance of contractors • Solution: 3 KPIs • • % of tickets with dissatisfied user feedback • % of tickets with a breached OLAs • % of tickets with a non-compliance report • Lessons learnt: • Start with a few, simple KPIs • Need to think about what to measure before writing the call for tender. 9 October 2018 ©CERN CC-BY-SA 4. 0 6

2. Bringing services on board Positive outcome examples Data Centre Hardware repairs CERN Hotel Operation (k. CHF) 6'000 5'000 4'000 3'000 2'000 1'000 2013 2014 2015 Revenues 9 October 2018 2016 2017 Charges ©CERN CC-BY-SA 4. 0 7

3. User experience Same process set for all tickets: No matter the interface: Service Portal, Service Desk, E-mail… the resulting ticket is the same • Simple process: easier to scale • Single process, catalogue, point of contact for users • Service. Now Service Portal Service Desk Service Management IT Facilities 9 October 2018 HR Finance ©CERN CC-BY-SA 4. 0 8

User experience Challenge: usage of mail still important, though decreasing Tickets per quarter, by report type 75 k Tickets by report type, relative 100% 50 k 42% 50% 25 k 36% 0% 2011 2012 2013 2014 2015 2016 2017 2018 0 9 October 2018 ©CERN CC-BY-SA 4. 0 9

3. User experience Incident vs Request Different processes are needed • But should have been hidden better from end users • • Would have been possible to fix after 2 -3 years (with significant work), but impossible now due to integrations • Ideal: unique identifier, no conversion between INC and RQF, field to differentiate and drive processes 9 October 2018 ©CERN CC-BY-SA 4. 0 10

3. User experience Online forms • Challenge: reduce back-and-forth communication between caller and supporter • Solution: lightweight online forms to request up front specific information from the caller • Possible thanks to fast development time with selected tool: 1 -5 days per form • Currently 450+ forms in the Portal and 100+ internal ones • Challenge: maintenance, upgrade to new Portal technology, usage of email / phone 9 October 2018 ©CERN CC-BY-SA 4. 0 11

3. User experience Ticket Feedback Before 9 October 2018 After ©CERN CC-BY-SA 4. 0 12

3. User experience Ticket Feedback 10 x more user feedbacks • 12% of users now give feedback on tickets • Results in continuous improvement of documentation, procedures, communication • Challenge: service quality evaluation, not only ticket handling evaluation. 9 October 2018 ©CERN CC-BY-SA 4. 0 13

3. User experience Knowledge Base: workflow simplification KB Submission KB Article Draft Functional Review Challenge: increase end Published user awareness and usage Layout Review Closed and Published Retired Closed and Rejected 9 October 2018 ©CERN CC-BY-SA 4. 0 14

3. User experience Service Status Board • • • Brings transparency to end users Intercepts incident creation Integrated with service catalogue Simple to edit Lesson learnt: simplify classification (ongoing) Challenges: increase end user awareness, notify of updates Service Availability overview: • Lesson learnt: show simple status (down / degraded / up vs. numerical value) and let service managers control it 9 October 2018 ©CERN CC-BY-SA 4. 0 15

4. Tool configuration Customise vs Extend vs Out of the box Customise Extend Out of the box • Modify what is • Build new functionality. • Do not modify or build provided out of the box Do not modify existing functionality • Some headstart, but • Easier upgrades • No upgrade trouble, upgrade trouble! but unsatisfied users and supporters! Lesson learnt: if you need to modify, extend instead of customise! 9 October 2018 ©CERN CC-BY-SA 4. 0 16

4. Tool configuration Customise vs Extend vs Out of the box Example: work management CERN Service. Now OOB Work Order Incident Request Parent ticket Work Task With new versions, Service. Now added or changed: Parent ticket (CERN) (OOB) Work Task Table Out ofcreated the boxat CERN Service. Now from scratch table 9 October 2018 Business logic engine • Triggers • Access permissions • New actions • New capabilities, e. g. auto dispatch depending on agent skills and location Each upgrade has required extensive testing and adaptation work. • ©CERN CC-BY-SA 4. 0 17

4. Tool configuration Integrations with other systems Motivations: support operations, increase value, reduce costs • Challenges: inter-team collaboration, communication, support and maintainability • Lessons learnt: • • If possible, implement one-way communication only (less risk) • When providing APIs, implement unit tests (e. g. GNI) • Two-way integrations: change control very important • Future: webhook system? 9 October 2018 ©CERN CC-BY-SA 4. 0 18

More topics: Change • User Interface evolution: new Service Portal • Problem • Work Management • SLAs • Data Privacy Protection / GDPR • CMDB • Data centre monitoring • 9 October 2018 ©CERN CC-BY-SA 4. 0 19

5. Conclusions 1 2 3 Common process and catalogue enables scalability Hide complexity from users Extend instead of customise your platform 9 October 2018 ©CERN CC-BY-SA 4. 0 20

Current and future challenges Current challenges: Data Privacy Protection / GDPR • New Service Portal: UI improvement for end users • Future challenges: Service Status Board enhancements (subscriptions) • Monitoring: automatic resolution of tickets • Raise awareness of some processes: problem, SLA / OLA… • AI for automatic classification, agent assist • 9 October 2018 ©CERN CC-BY-SA 4. 0 21

Thank you! Any questions? Many thanks to: - Barbara Brugger, Catherine Delamare, Eric Lienard, Isabel Fernández González, Mats Moller, Nicole Cremel, Olaf van der Vossen, Reinoud Martens, Zhechka Toteva - All other present and past members of the Service Management and Service. Now teams - All of our users david. martin. clavo@cern. ch snow-team@cern. ch 9 October 2018 ©CERN CC-BY-SA 4. 0