Thinking the unthinkable USING PROJECT RISK MANAGEMENT WHEN
Thinking the unthinkable: USING PROJECT RISK MANAGEMENT WHEN INTRODUCING COMPUTER-ASSISTED ASSESSMENTS Ian Harwood & Bill Warburton University of Southampton, UK.
Thinking the unthinkable… • Introduction • Overview of PRM and risk efficiency • Methodology / Case background • Identified risks • Conclusions
Introduction • Implementing CAA is a risky activity • Limited coverage in CAA literature • Aims of this paper: – Build links between CAA and PRM literature – Show formal PRM can be beneficial
Viewing CAA introduction as a project • Project has been defined as: 'a temporary endeavour undertaken to create a unique product or service' PMI (2000) • A project is: ‘a step into the unknown, fraught with risk and uncertainty’ Lock (2000)
Increasing interest in Risk • Now in the 'risk society' (Beck, 1992). • Broad base of risk literature covering: – environmental issues (GM foods, nuclear waste) – data processing – security management (terrorism, insurance) – project management – health and safety (design criteria, disaster planning) – finance and economics (derivatives, risk/return)
What is ‘Risk’? • ‘a choice rather than a fate’ Bernstein (1996) • risk or uncertainty? – risk is present when future events occur with measurable probability – uncertainty is present when the likelihood of future events is indefinite or incalculable Knight (1921) ‘Risk, Uncertainty and Profit’
Thinking the unthinkable… • Introduction • Overview of PRM and risk efficiency • Methodology / Case background • Identified risks • Conclusions
PRM interest groups • Association for Project Management: Risk Management Specific Interest Group (http: //www. eurolog. co. uk/apmrisksig/) • Centre for Risk Research, School of Management, University of Southampton, UK (http: //www. management. soton. ac. uk/risk/default. asp) • The Project Management Institute: Risk Management Specific Interest Group (http: //www. risksig. com/).
The Project Risk Analysis and Management process (Simon et al 1997)
Risk efficiency • From theory of portfolio management in Economics (Markowitz, 1959) • ‘Risk’ and ‘Return’ inextricably linked • Now used in the project environment – CCTA ‘Project Risk Management’ document
Theoretical risk efficiency framework
Risk efficiency migration due to PRM
Thinking the unthinkable… • Introduction • Overview of PRM and risk efficiency • Methodology / Case background • Identified risks • Conclusions
Overview of cases • Background to Soton MLE strategy • Roll out of Perception
Methodology • Case study approach (Yin, 2003) • Semi-structured interviews • Cross-case analysis (Miles and Huberman, 1994) • (retrospective) risk register generated
Thinking the unthinkable… • Introduction • Overview of PRM and risk efficiency • Methodology / Case background • Identified risks • Conclusions
Identified risks- before & during delivery • Wrong assessment published (C 3, 5) • Authors ‘reinvent the wheel’ (C 5) • Publication date missed (C 1) • Communication gaps (C 1, 3) • Environmental issues (C 4) • Cheating (C 1, 2, 3, 5, 6) • Failure at point of delivery- scalability issues (C 3, 4) • Tests submitted prematurely (C 2) Table 2
Identified risks- during & after delivery • Informal appeals- protest about fairness of CAA (C 1, 5) • Formal appeals: -workstation issues (C 1, 2, 6) -unfamiliar test environment (C 1) -test construction (C 1) -negative marking (C 6) -QA issues (C 5) CAA abandoned: -changes in course architecture (C 3) -staff turnover (C 6) -weak implementation (C 5) -perceived unfriendliness (C 5) Table 2
‘Proactive’ responses: mitigating actions 1 Table 2 • Pre-emptive Load-test at realistic levels • Allow sufficient time to fix the problem • Run a formative version of the test • Keep channels of communication open • Help staff to comply with internal CAA procedures • Ease of use- capture & address author feedback • Keep channels open between authors • Comply with internal CAA procedures • Use prefabricated templates
‘Proactive’ responses: mitigating actions 2 Table 2 • Make training a prerequisite for access to the service. • Help staff to comply with internal CAA procedures • Use random selections from large question pools • Allocate sufficient time to test workstations before exam • Use a class register to book sufficient workstations • Practice with formative versions • Maintain tutorials & practice tests on public workstations • Help tutors choose appropriate assessment methods
‘Reactive’ responses: Contingency Plans • Have a paper copy • Substitute marks from formative assessments • Capture and address author feedback • Discount compromised items • Re-issue test • Discount compromised items • Use trained invigilators • Have a spare workstation area available • Monitor and address student feedback Table 2
Thinking the unthinkable… • Introduction • Overview of PRM and risk efficiency • Methodology / Case background • Identified risks • Conclusions
Benefits of formal PRM • Enables better informed (and more believable) plans, schedules and budgets • Increases the likelihood of a project going to plan • Allows more meaningful assessment of contingencies • Contributes to the build-up of statistical information to assist in better management of future projects • Enables a more objective comparison of alternatives • Identifies, and allocates responsibility to, the best risk owner
Benefits of formal PRM cont… • Improves general communication • Leads to a common understanding and improved team spirit • Assists in the distinction between good luck / good management and bad luck / bad management • Helps develop the ability of staff to assess risks • Focuses attention on the real and most important issues • Facilitates greater risk-taking, thus increasing the benefits gained • Demonstrates a responsible approach to customers
Risk Management Maturity (Hillson, 1997) • Four levels: 1) naïve 2) novice 3) normalised 4) natural • Assists in identifying areas for improvement in risk management
Decay of risk exposure through iterative PRM
Closing thoughts… Six 'first time' applications of Perception replace paper-based exams- riskier than future repeat applications – iterative approaches to formalised project risk management can build institutional corpi of CAA risk knowledge – trend of total risk exposure progresses from risk inefficient to more risk efficient – a full-scale system could be released that already catered for the greatest risks – institution becomes more risk mature (Hillson, 1997)
Any questions…?
- Slides: 28