Performance Measurements of IMS systems Concepts 1 Copyright
- Slides: 33
Performance Measurements of IMS systems, Concepts 1 © Copyright 2008, Soft. Well Performance AB
Background to this presentation and proposal 2 © Copyright 2008, Soft. Well Performance AB
Background This was our message in a panel discussion at VON Rome A key to continued success (IMS) Quality needs to be defined and measured The involved standards (IMS) should include definitions of: • Quality metrics • Measurements of quality metrics • Evaluation and verification of measured quality figures Metrics should include: • Product characteristics • Production characteristics 3 © Copyright 2008, Soft. Well Performance AB
Background Why should performance testing have a standard? General reasons • We gain a general understanding of what should be regarded as performance • We will get a standard vocabulary for performance related issues • We will have one interpretation of every metrics • We will get a standard notation for Performance Test Cases • The “Babylon” of Performance testing will come to an end Practical reasons • Different tests that are performed according to the standard can be compared • Performance test tools can be certified • Performance test cases will be transferable between Test Tools that comply to the standard • Standardized Performance Test Case descriptions will be easier to maintain • Possible to define and certify Performance Test Suits • Will simplify regression testing (set-up and compare of results) The goal • To expand the notation of TTCN-3 to cover performance testing. This presentation is a short introduction to performance testing in general with extensions to IMS and its architecture. We hope to initiate a performance task. 4 © Copyright 2008, Soft. Well Performance AB
Performance Objectives & Performance Metrics 5 © Copyright 2008, Soft. Well Performance AB
Performance Objectives Three classes of performance objectives Measured Object Powerful Can the test object deliver what is required? Reliable Can the test object maintain services as required? Efficient Is the test object designed to be powerful and reliable? 6 © Copyright 2008, Soft. Well Performance AB
Performance Metrics describing powerfulness Delivery aspects Powerful Capacity Performance measurements of - Output (throughput) - Input capacity (peak load) - Concurrency (tx concurrently being processed) Speed of operation Measured Object Reliable Performance measurements of time for tested services - Response time (time to deliver response to a request) - Latency time (time to process or react) - Roundtrip time Scalability Efficient Performance measurements of improvements by - Adding more HW - Adding faster HW 7 © Copyright 2008, Soft. Well Performance AB
Performance Metrics (cont. ) Metrics describing reliability Carrier-Grade aspects Powerful Stability Has the tested system trends or patterns in Powerfulness and/or Efficiency over time? Availability Measured Object Reliable Unavailability in service delivery over time due to - Physical problems (Platform) - Logical problems (Application) Robustness Service levels during extreme conditions - Externally (Do. S attacks, peaks loads) - Internally (partial HW outage) Recovery Efficient Time to recover from - Partial or full restart of system - Update of system HW / SW Correctness Does the tested system deliver correct results under load? 8 © Copyright 2008, Soft. Well Performance AB
Performance Metrics (cont. ) Metrics describing efficiency Design aspects Powerful Resource usage Performance limits due to - Bottlenecks (design) - Too dynamic resource allocation (Peak-on-Peak) Resource utilization Measured Object Reliable Performance limits due to - Load distribution - Resource access (Queues) Resource balance Performance limits due to shortages in resources when there is plenty left of other resources Efficient Linearity & Scalability Performance limits due to - Non-linear processing - Multi CPU / Core design limits 9 © Copyright 2008, Soft. Well Performance AB
Performance Metrics (cont. ) Requirements on performance metrics Performance Metrics must be: - Uniquely identified ……………. . - Understandable ……………. . …. - Comparable. . …………………. . . - Repeatable. . . …………. - Accurate. . . …………. . . (metrics identifiers) (metrics specifications) (recording conditions and methods) (test case specs, test tool specs and test object specs) (data recording methods and configurations) 10 © Copyright 2008, Soft. Well Performance AB
Performance Metrics (cont. ) Performance metrics descriptions Measurement conditions - How ……. Metrics identifiers - What ……. - Where …. . - When …… Measurement Value(s) Metrics format - Value type - Unit - Accuracy (+/- %) Metrics types - Raw data - Normalized data - Derived data from … 11 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench Overview 12 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test bench surroundings Performance test specification Test Bench Measured Unit Measurement data and evaluation Powerful Reliable Efficient 13 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test bench components Test Bench Test Tool Clients Measured Unit Test Tool Servers Test Tool Clients Probes External Performance Data Internal Performance Data External Performance Data Measurement evaluation Powerful Reliable Efficient 14 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test specifications Performance test specification Test sizing Test bench configuration Test application (services) Test data specification Measurement specifications Test evaluation Test reporting 15 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test specifications (cont. ) Performance test specification Test sizing Test bench configuration Test application (services) Test sizing specifications Specify test duration How long time will the test run? Specify load Test data specification - Load levels - Load patterns - Load level variations Measurement specifications Specify simulated volumes - Number of simulated users - Amounts of test data Test evaluation Test reporting 16 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test specifications (cont. ) Performance test specification Test sizing Test bench configuration Test application (services) Test data specification Measurement specifications Test evaluation Test bench specifications Test bench components (physical & logical) - Test object components (SUT) - Test tool components Test bench transmission specs. - Used protocols - Addresses and ports for traffic - Load level variations Test bench traffic mapping - Mapping test tool components to test object components - Addresses and ports for traffic Test reporting 17 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test specifications (cont. ) Performance test specification Test sizing Test bench configuration Test application (services) Test application specifications (client side) Test application content (service profile) - What services should be requested Test application flow (traffic profile) Test data specification - In what order shall services be requested - Intelligent handling of different results of a service req. - Randomized order of service requests Measurement specifications Test application customization - Service request formats, multiple formats per service type - Individually modified service request messages Test evaluation Test reporting Test application processing specifications - Specification of processing flow of a service - Timer settings 18 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test specifications (cont. ) Performance test specification Test sizing Test bench configuration Test application (services) Test data specification Measurement specifications Test evaluation Test application specifications (server side) Test application flow (response profile) - Normal response mode - Negative response mode (errors, time-outs, disconnects etc. ) Test application customization - Service response formats, multiple formats per service type - Individually modified service response messages Test application processing specifications - Specification of processing flow of a service - Timer settings Test reporting 19 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test specifications (cont. ) Performance test specification Test sizing Test bench configuration Test application (services) Test data specifications User name specification User names to be inserted in messages - Client user name spaces - Server user name spaces User data specification User data variables to be inserted in messages Measurement specifications Test evaluation Test reporting 20 © Copyright 2008, Soft. Well Performance AB
Performance Test Bench overview Performance test specifications (cont. ) Performance test specification Test sizing Test bench configuration Test application (services) Test data specification Measurement specifications Test measurement specifications Measurement outside test object What should be measured at the test tools interfaces - Traffic rates, traffic volumes - Response time, latency time etc. Measurement inside test object What should be measured inside the test object - resources - measurement points (per server, per process, …) Test evaluation Test reporting 21 © Copyright 2008, Soft. Well Performance AB
Performance Measurement Methods 22 © Copyright 2008, Soft. Well Performance AB
Performance Measurement Methods Three performance measurement methods Load Simulation Monitoring Measured Unit When Pre Deployment Post Deployment Why Measuring limits Verify production req. Verifying requirements How Isolated functions Simple traffic Complex traffic What-if testing Active monitoring Passive monitoring 23 © Copyright 2008, Soft. Well Performance AB
IMS Performance Test Objects 24 © Copyright 2008, Soft. Well Performance AB
IMS Performance Test Bench Performance test bench components Test Bench Test Tool Clients IMS function(s) Test Tool Servers Test Tool Clients Probes External Performance Data Internal Performance Data External Performance Data Measurement evaluation Powerful Reliable Efficient 25 © Copyright 2008, Soft. Well Performance AB
IMS Performance Test Bench The IMS architecture (simplified) IMS is an architecture containing a large number of inter-acting functions, where each function performs a its tasks based on traffic over connected protocols. Overall performance is never better than the weakest function in the service chain, i. e. any function that does not match the performance of adjacent functions is a bottleneck or a risk. The IMS signaling plane HSS SLF 01 Diameter / Cx AAA services 03 Diameter / Sh 01 Diameter / Dx 03 HTTP / Ut Application services SIP-AS OSA-SCS IM-SSF 03 SIP / ISC Control services UE 01 SIP / Gm P-CSCF 02 SIP / Mw I-CSCF 02 SIP / Mw S-CSCF 02 SIP / Mw P-CSCF 01 SIP / Gm UE SGW 05 SIP / Mi 07 SIP / Mg 04 SIP / Mr ISUP / IP MGCF H 248 / Mn MGW 06 SIP / Mj BGCF The IMS media plane RTP MRF-C H 248 / Mp MRF-P 26 © Copyright 2008, Soft. Well Performance AB
IMS Performance Test Bench Performance testing IMS functions An integrated IMS system must be performance tested from individual functions, via sets of interacting functions to entire systems and interconnected systems A test bed for a single function may be quite complex to configure and set up. The IMS signaling plane Diameter / Cx Dx Test server Tools Traffic server Traffic Diameter / Sh / … AAA services Test server Tools Traffic server Traffic Diameter / Sh / … SIP / ISC HTTP / XCAP / Ut Application services SIP / Gm / Mw / ISC Test server Tools Traffic server Traffic SIP / MG / Mi / Mj / Mr Diameter / Cx Dx such as an such as a HSS or an SLF PTT server or a Presence server I-CSCF SBC MRF-C P-CSCF Control functions such as a or … BGCF 27 © Copyright 2008, Soft. Well Performance AB
IMS Performance Test Bench IMS function interfaces and requirements A test bed for a single function may be quite complex to configure and set up. IMS function Application protocols SIP Diameter XCAP other Transmission protocols TCP UDP SCTP other Interfaces Multiplexed Single users per port Individual IP address. 28 © Copyright 2008, Soft. Well Performance AB
IMS Performance Test Bench IMS function Test Tool interfaces Test Tools must have traffic units that match interfaces of all IMS components Test Tool Peers Test Tool Servers Server side Test Tool Servers Client side Test Tool Clients Server side Test Tool Bridges Test Tool Peers Server side Client side 29 © Copyright 2008, Soft. Well Performance AB
IMS Performance Test Bench IMS function Test Tool interfaces A Test bed may be set-up with several IMS functions interconnected by the test tool Test Tool Clients IMS Function Test Tool Servers Server side Client side IMS Function Client side Test Tool Peers Server side Test Tool Bridges Test Tool Peers IMS Function Server side Client side 30 © Copyright 2008, Soft. Well Performance AB
What is next, suggestions 31 © Copyright 2008, Soft. Well Performance AB
What is next, suggestions A standard for performance testing The track will contain a number of tasks • Create a proposal for performance related definitions and method • Create an abstract but precise notation for performance test cases • Create a standard for test tool components in a test bench • Create a proposal for language extensions to TTCN-3 First of all som practical activities • Investigate the interest for this and if yes • Set up a task force Thank you for your time 32 © Copyright 2008, Soft. Well Performance AB
“To measure is to know” - Lord Kelvin 33 © Copyright 2008, Soft. Well Performance AB