STF 374 Assessment of Quality of Service Qo

STF 374 Assessment of Quality of Service (Qo. S) via audits and surveys including the Customer Relationship Stages other than utilization of the service Qostic #35 October 29 th 2009 Pierre-Yves Hébert © ETSI 2009. All rights reserved

World Class Standards Introduction q Two deliverables to improve EG 202 009 "User Group; Quality of Telecom Services; Part 2: User related parameters on a service specific basis“ Ø DEG 202 843 "Method for testing the Qo. S parameters of the Customer Relationship Stages other than utilization" (on-going work) Define in detail a set of parameters and associated methods of testing which may be implemented by any stakeholder (Regulator, Service provider, consumers' organization …) Ø DTS 102 844 “Customer Relationship; Quality of Service (Qo. S) parameters assessment process; Requirements for bodies providing Conformity Assessment” (just starting) q A third one to define generic test suites Ø DTS 102 852 : User Group; Quality of Telecom Services; Assessment process of the Qo. S parameters of the customer relationship stages; Define a common set of test sequences that are applicable in order to provide results that are objective, reproducible and comparable INC - 2009 2

World Class Standards DEG 202 843 User Group; Quality of ICT Services; Definitions and Methods for assessing the Qo. S parameters of the Customer Relationship Stages other than utilization INC - 2009 3

World Class Standards Stages of the Service Customer Relationship Course q List of the stages of the Service Customer Relationship course Ø More than 80 parameters in total. Ø The intention is not to measure all of them continuously but to provide a methodology to ensure that when different stakeholders carry out Qo. S assessments according to this methodology, the results of these Qo. S measurements are fully reproducible, statistically valid and can be used to assess and compare the delivered Qo. S performance of Service Providers (SP). Ø The Guide does not cover the Qo. S of the telecommunication services themselves (already defined by other ETSI documents, e. g. EG 202 057 series, EG 201 765 series). Ø Thus the results could be used to compare the providers' performances over time or for benchmark. INC - 2009 4

World Class Standards Stages of the Service Customer Relationship Course q List of the stages of the Service Customer Relationship course (A timeline of these stages is available in annex) Ø Ø Ø Ø Preliminary information (PI) P 1 yz Contract establishment P 2 yz Service provisioning P 3 yz Service utilization (out of scope see EG 202 057 and EG 201 765 series) Service alteration P 4 yz Technical upgrade P 5 yz Service support, complaint management P 6 yz • • Ø Ø Documentation Technical support Commercial support Complaint management P 61 z P 621 -31 P 641 -52 P 661 -72 Repair services Metering, Charging, Billing Network/service Management by the customer Cessation INC - 2009 P 7 yz P 8 yz P 9 yz P 10 yz 5

World Class Standards Stages of the Service Customer Relationship Course q 4 Metrics: • Rate (%), e. g. • Time (t) e. g. P 104 = Time of PI delivery - Time of requesting PI • Number (N) e. g. where N=Number of potentially available modes of documentation; i=Index of each documentation mode; m= Actually available documentation modes • Opinion Rating (OR), mainly based on bipolar scales using seven steps • • • i = Index of expert N= Number of experts in the panel Where OR is the mean opinion rating, with ORi (i = 1…N) being the individual opinion ratings for the N members of the audit team. q General statement Ø Clause 4 provides the general statistical principles applying for this EG (this section refers also to TS 102 250 -6 published by STQ Mobile) Ø The assessments will make use of data from service providers, from surveys and assessments by experts. INC - 2009 6

World Class Standards Document structure (1/3) q Clause 5: Parameter Definitions Ø This part of the deliverable describes the definition of the parameters including parameter name, abstract definition and abstract equation. Ø It includes also a figure describing the timeline illustrating the relation between customer' point of view and the parameters • Note that in the figures diffferent boxes (and colours) are provided (see next slide) 5. 1 Customer Relationship Course: Preliminary information Ø 5. 1. 1 Parameter Name • 5. 1. 1. 1 Definition of parameter • It may include subclauses explaining in more details the parameter. • 5. 1. 1. 2 Equation • e. g. : • 5. 1. 1. 3 Measure • E. g. : Opinion Rating (OR) using a bipolar 7 point scale INC - 2009 7

World Class Standards Customer‘s point of view Parameter Legend q Blue boxes show what happens from the customer‘s perspective q No internal knowledge of operator processes available ● Green boxes show from which point of time to which point of time a certain action endures ● Here the according parameter number is mentioned (Pxxx) ● Dotted lines show the exact time reference for the specific parameter Parameter ● Boxes in dark green show provider centric parameters Parameter ● Already defined parameters (e. g. in EG 202 057 -1) STF 374 - Timelines 13. 07. 2009 INC - 2009 8 8

World Class Standards Document structure (2/3) q Clause 6: Evaluation Specific methodology/system • This part of the deliverable/document describes the methodology or the system which can be used to assess the parameter. Ø 6. 1 Customer Relationship Course: PI 6. 1. 1 Parameter Name (and metric) • 6. 1. 1. 1 Evaluation Specific description • Describe here the methodology or system which is used to assess the parameter (eg through a panel • 6. 1. 1. 2 Trigger Point Comparison • Trigger point from customer’s point of view The rating is carried out whenever a new service is being marketed and/or when significant changes are made to an existing service. • 6. 1. 1. 3 Accuracy of indicator (metric of the measure) • 6. 1. 1. 4 Representativeness • 6. 1. 1. 5 Presentation of parameter values INC - 2009 9

World Class Standards WP for future q For the time being each section includes the complete list of parameters as listed in EG 202 009 -2, including those whose assessment method is already defined in other standards; Ø However, some parameters have been modified (to be more correlated with the metric that may be measured Ø Some others could also be reviewed but this was considered out of the STF scope they are not detailled in the EG q Volunteers are invited to validate the Qo. S parameter testing (SPs, regulators) Ø Some positive answers have already been received, but the list is still open. INC - 2009 10

World Class Standards Document structure (3/3) q Annex A: Aggregate rating of a customer relationship stage (or performance category) from a set of individual performance parameter ratings Ø A first set of proposals have been included Ø EG 202 765 -1 shows that care should be taken when using an aggregate rating and is more in favour of displays as defined in ITU-T Recommendation P 505 INC - 2009 11

World Class Standards DTS 102 845 User Group; Customer Relationship; Quality of Service (Qo. S) parameters assessment process; Requirements for bodies providing Conformity Assessment INC - 2009 12

World Class Standards Draft content of the DTS 102 844 q DTS 102 844 “Customer Relationship; Quality of Service (Qo. S) parameters assessment process; Requirements for bodies providing Conformity Assessment” : Ø Conformity Assessment types Ø Certification type Ø Certification body • Certification Committee Ø Certification body staff • Auditor qualification criteria and competences Ø Application for certification • Information on the procedure • Preparation for audit Ø Audit • Audit report • Audit time • Monitoring INC - 2009 13

World Class Standards DTS 102 852 : User Group; Quality of ICT Services; Assessment process of the Qo. S parameters of the customer relationship stages INC - 2009 14

World Class Standards Scope of the DTS 102 852 q DTS 102 852 : User Group; Quality of ICT Services; Assessment process of the Qo. S parameters of the customer relationship stages : Ø This document will detail the test sequences based on the definition and test methods defined in DEG 202 043. The deliverable will define the generic test sequences for the 4 types of parameters, Time, percentage, number and Opinion Ratings. As an example it will also provide a set of dedicated test sequence for parameters of selected stages of the customer relationship course (Qo. S Assessment process). Ø The intention of this document is to define a common set of test sequences that are applicable by any stakeholder in order to provide results that are objective, reproducible and comparable. INC - 2009 15

World Class Standards Further plans q DEG 202 843 is posted on STF 374 docbox: Ø http: //portal. etsi. org/docbox/STF 374_USER_Qo. S_Assessment/Pu blic/DEG_202843 v 010101_clean. docx q STQ members have been invited to provide comments q DEG 202 843 has been reviewed by the User Group and approved as a stable draft, waiting for the results of the validation tests q Validation tests to start before the end of the year until end of March q At the same time extended consultation of stakeholders q Drafting of TS 102 844 and TS 102 852 starting soon q Final drafts for User Group review: April 2010 q Final drafts for UG approval by June 2010 q Workshop for presentation of the deliverables: Sept 2010 INC - 2009 16

World Class Standards Thanks Annex Detailled stages INC - 2009 17

World Class Standards Stage 1: Preliminary Information (PI) Customer‘s point of view Request for PI is sent PI is delivered Expected period of delivery (Timeout T 11) t 0 Parameter t 2 t 1 t 3 P 104: Response time P 101: Integrity P 103: Availability P 102: Pricing Transparency INC - 2009 18

World Class Standards Stage 2: Contract establishment (Version A) No customisation before contract, but option to customise after contract Optionally: Changed Change request contract (after received signature) sent Timeout T 22 Contract is received, signature of contract without changes 0 t 3 P 201: Integrity of contract information P 202: Compliance of contract with PI t 4 t t 5 P 204: Ease and flexibility to amend terms after formal contract INC - 2009 P 201: Integrity P 202: Compliance of contract with PI 19

World Class Standards Stage 2: Contract establishment (Version B) Customistation before contract, but option to customize after contract Contract proposal received 0 t 1 Custom. Final contract is ised received, contract signature of sent contract Timeout T 21 t 2 P 203: Flexibility of contract customization Optionally: Changed Change request contract (after signature) received sent Timeout T 22 t 4 t 3 P 201: Integrity of contract information P 202: Compliance of contract with PI INC - 2009 t 5 P 204: Ease and flexibility to amend terms after formal contract t P 201: Integrity P 202: Compliance of contract with PI 20

World Class Standards Stage 3: Service provisioning (fixed date) Provisioning date received 0 Provisioning Promised Datedone t 2 t 1 P 306: Punctuality of appointment t P 301: Rate of meeting promised provisioning date P 304: Rate of contract cancelled due to non-fulfillment of contract P 302: Time for provisioning P 305: Complet. of fulfilment of contractatual specification in prov. of service P 308: Provisioning not complete and correct first time INC - 2009 21

World Class Standards Stage 3: Service provisioning (period) Provisioning period received Appointment Provisioning done Provisioning period (Timeout T 31) 0 t 1 t 2 t t 3 t 4 P 306: Punctuality of appointm. P 302: Time for provisioning P 303: Rate of successful prov. within specified period INC - 2009 P 304: Rate of contract cancelled due to nonfulfillment of contract P 305: Completion of fulfilment of contractatual specification in prov. of service P 308: Provisioning not complete and correct first time 22

World Class Standards Stage 3: Service provisioning (punctuality) Appointment made OR equipment delivery announced 0 t 1 Too early In time Too late Appointed date/time Scheduled equipment delivery t 2‘‘ or • t P 306: Punctuality of appointments P 307: Punctuality of equipment delivery P 306 and P 307 calculates the relative delay between the announced point of time for equipment delivery or provisioning and the effective point of time INC - 2009 23

World Class Standards Stage 4: Service alteration Alteration period received Appointment Alteration done Alteration period (Timeout T 41) 0 t 1 t 2 t 3 P 404/P 405 Punctuality of appointm. Reliability period (Timeout T 42) t 4 t t 5 P 408: Technical reliability of serv. within an agreed period after alt. P 403: Completeness of fulfilment of contractual. specification P 406: Alteration not complete and correct first time P 402: Rate of successful service P 407: Conformity and success rate of alteration within specified period service alteration P 412: Organisational efficiency of the SP to carry out service alteration P 401: Time for alteration INC - 2009 24

World Class Standards Stage 4: Service alteration (punctuality) Appointment made OR equipment delivery announced 0 t 2 Too early In time Too late Appointed date/time Scheduled equipment delivery t 3‘‘ or • Same concept as in Stage 3 INC - 2009 t P 404: Punctuality of appointments P 405: Punctuality of equipment delivery 25

World Class Standards Stage 5: Technical upgrade I Upgrade period received Outage begins Appointment Upgrade done Upgrade period (Timeout T 51) 0 t 1 t 2 Outage ends Outage t 3 P 504: Punctuality of appointm. P 501: Time for upgrade of a service P 502: Rate of successful upgrade within specified period t 4 Reliability period (Timeout T 52) t 5 t 6 t P 505: Outage time due to upgrade P 503: Completeness of fulfillment of specification P 512: Organisational efficiency P 513: Competence and preparedness INC - 2009 26

World Class Standards Stage 5: Technical upgrade II Upgrade period received Appointment Outage begins Upgrade done Upgrade period (Timeout T 51) 0 t 1 t 2 t 3 Outage t 4 Outage ends Reliability period (Timeout T 52) t 5 t 6 t P 508: Technical reliability of service P 506: Technical upgrade not complete and correct first time P 507: Conformity and success rate of technical upgrade P 512: Organisational efficiency P 513: Competence and preparedness INC - 2009 27

World Class Standards Stage 5: Service upgrade (punctuality) Appointment made OR equipment delivery announced 0 t 2 Too early In time Too late Appointed date/time Scheduled equipment delivery t 3‘ t 3 t t 3‘‘ P 504: Punctuality of appointments • Same concept as in Stage 3 INC - 2009 28

World Class Standards Stage 6 a: Service Support – Documentation Service provisioned Documentation received A) Initial provisioning Service changed Adapted doc. received B) Update procedure Documentation delivery period (Timeout T 61) 0 t 1 t 2 t t 3 P 611: Documentation delivery time P 613: Integrity of documentation P 612: Availability of documentation within specified period P 615: Legibility of documentation P 614: Modes of documentation P 616: Overall reliability INC - 2009 29

World Class Standards Stage 6 b: Service Support – Technical Support Problem occurred, try to contact support Contact established Solution proposal applied Reliability checked Problem description given Timeout T 62 0 t 1 P 621: Rate of accessibil. to technical support Solution proposal received Stability period (Timeout T 64) Solution period (Timeout T 63) t 2 P 627: Rate of recognition of costumer request t 3 t 4 t 2‘ P 628: Response time of tech. support P 622: Solutions resolved within specified period t 2 can be equal to t 3, e. g. for mail exchange or web access INC - 2009 t t 5 t 6 P 625: Rate of reliability P 623: Number of attempts before successful solution P 624: Integrity of solution P 626: Modes of tech. support 30

World Class Standards Stage 6 c: Service Support – Commercial Support Need for change occurred Timeout T 65 Contact established Solution proposal received Need for change described Solution period (Timeout T 66) t 0 t 1 t 2 t 3 ‘ P 642: Solution delivery time t 4 P 641: Rate of accessibility to comm. support P 643: Rate of solutions P 647: Response delivered within specified time of the comm. period support P 652: Organisational efficiency INC - 2009 t 5 P 644: Integrity of solution P 645: Modes of commercial support 31

World Class Standards Stage 6 d: Complaint Management (CM) Complaint reason occurred CM contacted Timeout T 67 t 2 0 t 1 t 2 P 661: Rate of ‘ accessibi-lity to CM Complaint accepted Complaint filed Timeout T 68 Solution delivered Solution period (Timeout T 69) t 3 t 4 t 6 t 5 P 665: Integrity of P 662: Rate of P 669: Customer complaint resolution complaint recognit. P 666: Customer resolution time P 663: perception of CM Solution not P 664: Rate of complete and solutions delivered within specified period correct first time t P 668: Response time of CM desk P 667: Overall Quality of the CM process P 672: Organisational efficiency INC - 2009 32

World Class Standards Repair request sent Stage 7: Repair Services (RS) Request accepted Repair period announced Timeout T 72 Repair period (Timeout T 73) Timeout T 71 0 t 1 P 701: Accessibility to RS t 2‘ Repair done t 3 Reliability period (Timeout T 74) t 4 P 702: Successfull repairs caried out within a specified period P 706: Fault repair time P 704: Punctuality t 5 t 6 t P 703: Rate of successful first time repairs P 705: Efficiency of the repair service P 707: Frequency of customer complaints related to RS P 708: Prof. / P 709: Ability SP P 710: User friendliness of RS P 711: Organisational efficiency INC - 2009 33

World Class Standards Stage 8: Metering, Charging, Billing Access to documents enabling for expense control Access to expense control information in chosen mode Document found Timeout T 81 0 t 1 t 2 t P 801: Rate of accessibility to tariff information INC - 2009 34

World Class Standards Stage 8: Metering, Charging, Billing Expense overrun notification Billing budget overrun notification received Billing budget overrun occurred Timeout T 82 0 t 3 P 802: Rate of successful notification of exceeding billing budget t 4 t P 803: Notification delay of exceeding billing budget INC - 2009 35

World Class Standards Stage 8: Metering, Charging, Billing Real-time access to expense information Access to account information Real-time account information received Timeout T 83 0 t 5 P 804: Rate of accessibility to the account management P 805: Time delay of the charging information t 6 INC - 2009 t 36

World Class Standards Stage 8: Metering, Charging, Billing Bill production and delivery Bill expected Bill received Timeout T 84 t 0 t 7 t 8 P 809: Modes of P 806: Rate of received billing information bills transfer P 807: Bill delivery delay P 815: Organisational efficiency of billing service INC - 2009 37

World Class Standards Stage 8: Metering, Charging, Billing Bill production and delivery after direct debit Direct debit Bill expected Bill received applied Timeout T 85 0 t 9 t t 11 t 10 Definitions: P 808: Rate of bill P 809: Modes of availability after direct billing information debit transfer P 815: Organisational efficiency of billing service INC - 2009 A) Direct debit: Variable amount B) Standing order: Fixed amount 38

World Class Standards Stage 9: Network/Service Management by the customer Access outages: Access to network mgmt facilities is not possible Access outage begins 0 t 1 tout 1 Access outage ends Access outage begins toutn tout 2 Access outage ends toutn+1 t 2 t P 901: Outage duration P 902: Frequency of outages • P 901: Time parameter INC - 2009 39

World Class Standards Stage 9: Network/Service Management by the customer Customer request sent Reaction to customer request observed Timeout T 91 0 t 1 P 903: Response time for reply to requests P 904: Request response rate t t 2 P 906: Rate of accessibility to N/S management facility P 907: Response time of P 908: Access time the operator of N/S mgmt. P 910: Overall quality of the N/S management process P 911: Provider ability match customer wishes P 905: Overall reliability of network/service management P 913: Organisational efficiency of the network/service management service P 909: Frequency of customer complaints P 912: User friendliness INC - 2009 40

World Class Standards Stage 10: Cessation request sent Acknowledgement to cessation request observed Timeout T 101 Cessation is completed Timeout T 102 t 0 t 1 t 2‘ t 3 P 1001: Cessation acknowledgement time P 1002: Cessation acknowledgement rate P 1003: Rate of accessibility to cessation facility P 1004: Rate of contractual cessations achieved INC - 2009 41

World Class Standards Examples of graphic display of the Qo. S assessments INC - 2009 42

World Class Standards Complaints Internet SP A V 1 Preliminary information 5. 2 Contract establishment Cessation 15 12. 9 16 40 80 11 9 31 27 60 7 42. 6 40 22 Service provisioning 3 15. 260 40 30. 4 13 20 0 0 10 4 40 53. 1 70 7 40 9. 7 Service operation 6. 1 7 100 17. 7 Repair services 11. 5 16 10 8018. 2 46. 1 60 40 10 -2 19. 4 20 30 20 -11 -1 0 -10 -20 1 0 Metering-Charging-Billing 20 25 3. 5 Commercial support 6. 9 Technical support INC - 2009 43

World Class Standards Complaints Internet SP A V 2 Preliminary information 5. 2 Contract establishment Cessation 0 12. 9 16 0 9 18 28 12 27 42. 6 56 36 Service provisioning 15. 2 0 0 6 18 54 20 30. 4 40 60 84 39 0 18. 2 31 62 46. 1 93 112 80 24 72 140 16 105 12 26 Metering-Charging-Billing 24 70 19. 4 13 53. 1 35 16 8 0 9. 7 Service operation 6. 1 0 17. 7 Repair services 11. 5 8 4 0 0 3. 5 Commercial support 6. 9 Technical support INC - 2009 44

World Class Standards Complaints Internet SP B V 1 Preliminary information 13. 7 Contract establishment Cessation 15 36. 5 74. 1 40 80 11 9 31 60 7 27 42. 6 40 22 Service provisioning 58. 360 13 40 30. 4 20 0 0 -11 -1 -10 -20 1 0 10 Metering-Charging-Billing 20 3 20 0 10 -2 40 19. 4 20 53. 1 70 4 30 7 40 34. 3 Service operation 6. 1 7 10 8079. 6 60 4046. 1 100 91. 7 Repair services 11. 5 16 25 8. 5 21. 8 Commercial support Technical support INC - 2009 45

World Class Standards Complaints Internet SP B V 2 Preliminary information 13. 7 Contract establishment Cessation 0 36. 5 74. 1 0 9 18 28 12 27 42. 6 56 36 Service provisioning 58. 3 0 6 18 84 54 0 20 30. 4 40 60 39 0 31 62 46. 1 93 112 80 24 72 140 16 79. 6 105 12 26 Metering-Charging-Billing 24 70 19. 4 53. 1 13 35 16 8 0 34. 3 Service operation 6. 1 0 91. 7 Repair services 11. 5 8 4 0 0 8. 5 Commercial support 21. 8 Technical support INC - 2009 46

World Class Standards Complaints Internet SP C V 1 Preliminary information 11 Contract establishment Cessation 15 39. 6 67. 4 40 80 11 9 31 60 7 27 42. 6 40 22 Service provisioning 3 40. 160 13 40 30. 4 20 0 10 -2 40 19. 4 20 53. 1 70 4 30 7 40 26. 1 Service operation 6. 1 7 8071. 8 60 4046. 1 20 -11 -1 0 -10 -20 1 10 Metering-Charging-Billing 20 100 91. 4 Repair services 11. 5 16 10 25 9. 6 14 Commercial support Technical support INC - 2009 47

World Class Standards Complaints Internet SP C V 2 INC - 2009 48

World Class Standards Complaints Internet SP D V 1 Preliminary information 4. 4 Contract establishment Cessation 15 21. 1 52. 2 40 80 11 9 31 60 7 27 42. 6 40 22 Service provisioning 13. 360 13 40 30. 4 20 0 0 -11 -1 -10 -20 1 0 10 Metering-Charging-Billing 20 3 20 0 10 -2 40 19. 4 20 53. 1 70 4 30 7 40 16. 7 Service operation 6. 1 7 8046. 7 60 4046. 1 100 40 Repair services 11. 5 16 10 25 5. 6 3. 3 Commercial support Technical support INC - 2009 49

World Class Standards Complaints Internet SP D V 2 INC - 2009 50

World Class Standards Provisioning P 301 60 P 302 100 8 90 15 10 9 97 20 10 7 70 6 80 100 8 30 80 12 P 303 P 309 0 P 308 -10 60 -20 95 94 3 50 88 -20 -30 91 -1 -10 1 10 20 99 10 -2 5 10 P 304 20 10 -7 98 0 5 0 8 0 0. 15 100 3 1 P 307 1 95 P 305 0. 2 P 306 INC - 2009 51

World Class Standards Thanks again … INC - 2009 52