IBM T J Watson Research Center Northwestern University
IBM T. J. Watson Research Center & Northwestern University Towards an understanding of Decision Complexity in IT Configuration Bin Lin Department of Electrical Engineering & Computer Science, Northwestern University binlin@cs. northwestern. edu Aaron Brown IBM T. J. Watson Research Center abbrown@us. ibm. com © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Context: Quantifying IT Process Complexity § Technical problem – Identify metrics and develop methodology for quantifying the exposed operational complexity of IT processes § Importance – Complexity of systems management processes drives labor cost – Labor cost reductions are extremely important to services/outsourcing organizations and customers – A quantitative framework for complexity can guide process improvements to reduce labor cost – Opportunities for deploying autonomic computing in IT environment 2 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Previous work § Initialize a model of configuration complexity and demonstrates its value for a change management system. § Metrics that indicate some configuration complexity, including execution complexity, parameter complexity, and memory complexity. § Example: complexity of J 2 EE provisioning § Process complexity: manual • Execution 1 595 steps, 27 context switches • • Parameter • • • automated 17 17 32 parameters used 61 times, 018 outside of source context Source score: 125 94 Memory (LIFO stack model) • 0 0 Size: max 8, avg 4. 4 See: Brown, A. B. , A. Keller, and J. L. Hellerstein. A Model of Configuration Complexity and Its Application to a Change Management System. Proceedings of the Ninth IFIP/IEEE International Symposium on Integrated Network Management (IM 2005), Nice, France, May 2005. 3 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Next Step: Decision Complexity § Previous metrics assume expert skill – Do not consider complexity arising from decision-making Procedure Design Space s § Capturing complexity impact of decisions along a specific procedure’s path – Parameterized by skill level goal § Understanding the overall complexity across all possible procedures § Quantifying the tradeoff between flexibility and simplicity vs. s 4 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Decision Complexity (An initial model & methodology) § Factors that affect complexity – constraints e. g. compatibility between software products, capabilities of a machine consequences e. g. functionality, performance levels of guidance e. g. documentation, previous configuration experience Install/Config Procedure for J 2 EE App Need Enterprise Clustering ? Y N Install Cloudscape + WAS Express § Manifestation – task time, user-perceived difficulty, error probability § A starting point to drive data collection (user study) . . . Install DB 2 UDB + WAS ND § After we have the real-world data, refine the model 5 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Model details: levels of guidance § Global information – E. g. documentation, design guide, deployment patterns § Short-term goal-oriented information – E. g. wizard-based prompts indicating the appropriate next step § Confounding information – E. g. alternate configuration instructions for a different platform than the target § Position information – E. g. feedback on the current state of the system and the effect of the previous action 6 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Decision Complexity (challenge & solution) § Hard to conduct a full user study to validate the model (constraints, consequences, levels of guidance) using real IT processes § Sol: measuring decision complexity in a simplified domain: Route-planning – navigating a car from one point to another 7 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Decision Complexity (user study design) § Web-based study – larger subject pool – accurate timing data – standardized information § Questionnaire to collect user background § Recording user interaction – time spent, each decision point – comparison b/w user path & optimal path – user ranking of the complexity for testcases 8 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Testcase selection § Testcases – Different combinations of factors • • • Static traffic Dynamic traffic Expert path GPS Difference in travel times Position information – Selected 10 most relevant testcases Expert path – Example: dynamic traffic (speed updates) + expert path 9 Towards an understanding of Decision Complexity in IT Configuration Dynamic traffic © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University User Study: Overview & Analysis Approach § Overview – 3 experiments, 10 testcases with 1 warm-up – 1 st stage, 35 users – 2 nd stage, 23 users, with refined experiment § Metrics – Average time spent per step (e. g. time / no. of steps) – User rating (in the end of each experiment) – Error rate (user picked non-optimal path) § Analysis approach – Step I: general statistical analysis of all data • Each testcase measured as an independent data point • Goal: identify factors that explain the most variance – Step II: pair-wise testcase comparisons • Get more insight into specific effects of factor value • Goal: remove inter-user variance 10 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Summary of results § Significantly different impacts on user-perceived difficulty than on objective measures (e. g. time and error rate) § Time is influenced by: – Constraints • static constraints > dynamic; static constraints > without constraints – Guidance (goal) • without short-term goal oriented guidance > with such guidance § Rating is influenced by: – Guidance (goal) – Guidance (position) • without position guidance > with such guidance – Constraints • static constraints > dynamic § Error rate: hard to say statistically, except – error rate is reduced when guidance (goal) is present – error rate is reduced when guidance (position) is not present 11 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Summary of results (cont) – Depending on its goal (user, time or error rate), optimization for less complexity will have different focus, examples: • An installation procedure with easily-located clear info (e. g. wizardbased prompts) for the next step will reduce both task time and user-perceived complexity • A procedure with feedback on the current state of the system and the effect of the previous action (e. g. message windows following a button press) will only reduce user-perceived complexity, but unlikely to improve task time or error rate • Omitting positional feedback (i. e. , not showing users effects of their actions) may, counterintuitively, increase user accuracy, but at cost of significantly higher perceived complexity and task time 12 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Proposal for a new user study • Validate the model in the IT configuration domain 13 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Analogy between two studies 14 • Driving time per segment • Number of features achieved per step • Global map • Flowchart of the overall process (text) • Traffic • Soft compatibility / machine capacity limit • Goal (reach the destination) • Achieve the max number of features Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Further step • Apply the model to assess IT decision complexity Complexity (Constraints, Guidance, Consequence, …) 15 Avg. Time. Per. Step Operation time Rating (User perceived complexity) Skill levels Error Rate Probability (downtime) Towards an understanding of Decision Complexity in IT Configuration Cost ($) © 2006 IBM Corporation
IBM T. J. Watson Research Center & Northwestern University Conclusions § We investigated decision complexity in IT configuration procedures – Used an carefully-mapped analogous domain to explore complexity space – Conduct an extensive user study – Quantitative results showing the key factors – Some guidance for system designers seeking to reduce complexity – Next steps are to explore further in simulated IT environment 16 Towards an understanding of Decision Complexity in IT Configuration © 2006 IBM Corporation
- Slides: 16