Developing Maintaining Dynamic MicroSimulation Models at DWP Sally

Developing & Maintaining Dynamic Micro-Simulation Models at DWP Sally Edwards Simon Gault ESRC – 2 nd April 2009 1 Model Development Unit Strategy Directorate

Developing Models at DWP 1. Overview 2. Objectives 3. Development Standards 4. Simplifying Complexity 5. Maintenance Protocol 6. What worked well and Lessons Learned 2 Model Development Unit Strategy Directorate

Introductions Simon Gault Ø Statistician, with a strong background in Economics Ø Extensive experience in developing models Ø Manages the Forecasting Simulation models team Sally Edwards Ø Systems / Business Analyst Ø Predominantly worked in the private sector on large scale IT systems Ø Manage the Genesis Simulation Model Members of a the Model Development Unit (MDU) which has responsibility for Micro-Simulation Models at DWP MDU consists of 20+ people 3 Model Development Unit Strategy Directorate

Micro-Simulation Models Ø Dynamic Micro-Simulation Models, based on our standard architecture, Genesis: Ø Pensim 2 private & state pension income Ø Inform Integrated Forecasting model for working age benefit claimants Ø Individual Benefit Forecasting for RP, IB, DLA/AA Ø Employment Model, currently being validated Ø Static Micro-Simulation Models: Ø Policy Simulation Model (PSM) Ø Employment and Hours Model (EHM) 4 Model Development Unit Strategy Directorate

Key factors taken into Consideration during Development Key factors considered when we designed our first dynamic micro-simulation model, Pensim 2 - also valid for our other models: Ø Simplify the complexity of the model as much as possible, using a modular approach and allowing a simple prototype model to be developed quickly Ø Several developers work on each model at the same time Ø Large staff turnover Ø Models are intended for long term use (e. g. Pensim 2 anticipated to have 15 -20 year life) so most of the people using/maintaining the models are not involved in their development Ø Clear documentation, processes, training and user guides are essential Ø Excel and SAS are the preferred tools, as both are well understood and extensively used among the Analysts at DWP 5 Model Development Unit Strategy Directorate

Developing Models at DWP 1. Overview 2. Objectives for Pensim 2 (first DMS model) 3. Development Standards 4. Simplifying Complexity 5. Maintenance Protocol 6. What worked well and Lessons Learned 6 Model Development Unit Strategy Directorate

Original Objectives for Pensim 2 Feasibility Study document (2002) – Objectives 1. Flexibility 2. Robustness 3. Transparency 4. User-friendliness 5. Speed 6. Reliable Output 7. Availability (i. e. desktop model) 8. Efficiency (in use of memory) 9. Timeliness 10. Done with own expertise 11. Done with own resources 12. Ease of handover 13. Ease of maintenance 14. Independence of base data 7 Model Development Unit Strategy Directorate

Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: 8 Model Development Unit Strategy Directorate

Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: Excel based front end – Pensim 2 § User friendly, transparent code in a standard format § Easy to understand flexible, enabling analysts to change parameters & structure of model without understanding the underlying code. § Easy to maintain and handover § Allowed independence of base data 9 Model Development Unit Strategy Directorate

Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: : Excel based front end – Pensim 2 § User friendly, transparent code in a standard format § Easy to understand flexible, enabling analysts to change parameters & structure of model without understanding the underlying code. § Easy to maintain and handover § Allowed independence of base data SAS code generator - Genesis § Developed with our own expertise and own resources § Robust generator, producing reliable outputs 10 Model Development Unit Strategy Directorate

Basic structure of Genesis Dynamic Micro-Simulation Models at DWP 11 Model Development Unit Strategy Directorate

Developing Models at DWP 1. Overview 2. Objectives 3. Development Standards 4. Simplifying Complexity 5. Maintenance Protocol 6. What worked well and Lessons Learned 12 Model Development Unit Strategy Directorate

Development Standards Two separate development teams, with different development standards adopted Genesis Model Engine Ø Developed by IT staff Ø Used standard IT project development procedures Pensim 2 Analytical team Ø Economists and Statisticians Ø Less structured approach to development 13 Model Development Unit Strategy Directorate

Genesis Model Engine Development Standards Highly structured approach to the development Ø Project Management Protocol based on Prince 2 methodology, although not all aspects were needed Ø Documentation – 2 types Ø Development documentation produced at each stage of project, that was signed off and used as input to the next stage Ø Key Project documentation (*) - maintained and kept up-to-date with each new release Ø Development Processes –simple process diagrams to provide clear guidelines, particularly useful for new staff Ø Detailed Requirements – explaining the full functionality of the model engine. Document translates Economist language into IT language (*) 14 Model Development Unit Strategy Directorate

Genesis Model Engine Development Standards Ø Design Diagrams and documentation – used in development stage only Ø Change Control process –changes estimated and impacted before being accepted/rejected/on hold (*) Ø Problem Log process –problems recorded, prioritised, fixed/removed from the scope/user error (*) Ø Validation Process - detailed Test Plan based on Requirements document, used to ensure that each requirement was fully satisfied Ø Test Pack ensured existing functionality continued to work when changes were made to the model engine (*) Ø User Guide contains details of functionality written in simple format (*) Ø Training materials, inc presentations and training text for self-study (*) 15 Model Development Unit Strategy Directorate

Pensim 2 Model Development Ø Pensim 2 Analytical development followed a less structured approach Ø Generally this was appropriate, as detailed requirements were not possible to establish up-front Ø Effort was predominantly spent on regression analysis Ø Analysis results reviewed to determine the key components to include and the most appropriate sources for the assumptions Ø Using the standard Genesis templates, the Pensim 2 model was more easy to develop, rather than if a traditional DMS model had been built 16 Model Development Unit Strategy Directorate

Developing Models at DWP 1. Overview 2. Objectives 3. Development Standards 4. Simplifying Complexity 5. Maintenance Protocol 6. What worked well and Lessons Learned 17 Model Development Unit Strategy Directorate

Key Aim of Genesis Structure - Simplify Complexity No Hard Coding Rule which allows complete flexibility for: • Structure of the simulation, i. e. order of events processed • Data structures, i. e. data tables, data relationships and variables • Parameters, e. g. indexes, rates, dates • Regression Equations • Selection filters • Static SAS code, if required (e. g. addition of new cases to a model) • Ability to add / remove modules easily Each model is defined in simple Excel spreadsheets, using standard templates, that users can easily understand modify 18 Model Development Unit Strategy Directorate

Key Aim of Genesis Structure - Simplify Complexity Data Access routines Ø Enable easy access to data so user does not have to write complex code Ø Default data access will process the current object in the current simulation year Ø Facility to process a variable belonging to a related object, e. g. partner PAR; Relationships are not hard-coded and new relationships can easily be defined Ø Data manipulation routines provide commonly required data facilities, e. g. “maximum value during past 10 years” = MA 10 Ø E. g. MA 10_PAR_pa_salary returns the maximum value in the past 10 years of the salary variable for the individual’s partner, taken from the pa table Keep Equations simple Ø Equations, filters & process flows are defined in a simple, standard format, that is unambiguous. Hence the models are easy to maintain 19 Model Development Unit Strategy Directorate

Developing Models at DWP 1. Overview 2. Objectives 3. Development Standards 4. Simplifying Complexity 5. Maintenance Protocol 6. What worked well and Lessons Learned 20 Model Development Unit Strategy Directorate

Change Management Ø Change Management Process is followed for all models Ø Every change goes through the formal Change Control procedure, i. e. requirements documented, impact assessed, agreed by users & signed off Ø Change Register listing all changes, each with a separate reference number Ø For each change to the model, a separate Change Control form is produced Ø The model code contains the Change ref number, with a comment Ø Some changes are repeated regularly, e. g. assumptions, so clear documentation is helpful for the repeat change Ø Questions arise after a change has been implemented - these can be easily answered by referencing the Change Control documentation 21 Model Development Unit Strategy Directorate

Problem Management Ø Problem Management Process is followed for all models Ø Every problem/error discovered in a model goes through the formal Problem Log procedure, i. e. requirements documented, impact is assessed, agreed by users & signed off Ø Problem Log Register, listing all problems identified, including those that turn out to be User Errors or misunderstandings; PL ref number assigned Ø For each problem recorded, a separate Problem Log form is produced Ø Users are notified of significant problems that affect the model results Ø Many of the problems are minor and are picked up by the development team – these are recorded formally, even if they do not impact the results 22 Model Development Unit Strategy Directorate

User Model Ø Our models fall into 2 categories: Ø Maintained & owned by the Model Development Unit Ø Developed by Model Development Unit; handed over to User team, who then own and maintain the model Ø Models owned by the Model Development Unit have a User Group and/or a Steering Group consisting of representatives from each team using the model Ø The purpose of these groups is to agree the content, priority and timing of changes made to the model 23 Model Development Unit Strategy Directorate

User Group Purpose Specifically for Pensim 2, our largest model: Ø Change Requests and Problem Logs are reviewed by the User Group, where appropriate Ø A key user sponsor is assigned to each change. The sponsor ensures that the change is accurately specified & will sign-off testing Ø The User Group are responsible for signing off new versions of the model before they are released Ø The users also have a separate User Forum where they present analysis carried out on the model outputs and explain how the results have been used 24 Model Development Unit Strategy Directorate

Release Management Again, specifically for Pensim 2: Ø New Releases are implemented approximately every 6 to 9 months Ø Each Release includes one or two key components, e. g. Private Pension Reform, and a bundle of smaller changes and problem fixes Ø The timing and content of each Release are agreed with the User Group Ø When a new Release is built: • Each change/fix is added to the model one at a time • Hence each change is reviewed and signed off separately • Enables us to fully understand agree the effect of each change. • Time consuming but gives confidence in the results Ø Clear audit trail Ø All previous versions of the model remain available 25 Model Development Unit Strategy Directorate

Developing Models at DWP 1. Overview 2. Objectives 3. Development standards 4. Simplifying Complexity 5. Maintenance protocol 6. What worked well and Lessons Learned 26 Model Development Unit Strategy Directorate

Original Objectives Satisfied Pensim 2 Feasibility Study document (2002) – Objectives 1. Flexibility 2. Robustness 3. Transparency 4. User-friendliness 5. Speed 6. Reliable Output 7. Availability (i. e. desktop model) 8. Efficiency (in use of memory) 9. Timeliness 10. Done with own expertise 11. Done with own resources 12. Ease of handover 13. Ease of maintenance 14. Independence of base data 27 Model Development Unit Strategy Directorate

Objectives Partially Satisfied Pensim 2 Feasibility Study document (2002) – Objectives 1. Flexibility 2. Robustness 3. Transparency 4. User-friendliness 5. Speed 6. Reliable Output 7. Availability (i. e. desktop model) 8. Efficiency (in use of memory) 9. Timeliness 10. Done with own expertise 11. Done with own resources 12. Ease of handover 13. Ease of maintenance 14. Independence of base data 28 Model Development Unit Strategy Directorate

Objectives that proved more tricky Pensim 2 Feasibility Study document (2002) – Objectives 1. Flexibility 2. Robustness 3. Transparency 4. User-friendliness 5. Speed 6. Reliable Output 7. Availability (i. e. desktop model) 8. Efficiency (in use of memory) 9. Timeliness 10. Done with own expertise 11. Done with own resources 12. Ease of handover 13. Ease of maintenance 14. Independence of base data 29 Model Development Unit Strategy Directorate

What went well Ø Generic framework has proved successful and easy to use Ø Genesis architecture has been remarkably robust Ø Modules have been reused/copied from one model to another Ø Quicker development of new dynamic micro-simulation models, compared with a traditional build Ø Automated documentation facility and error checking produced using VBA, listing where variables are used and providing an overview of each module, to ensure consistency Ø Lessons learnt from Pensim 2 development were used when developing subsequent models 30 Model Development Unit Strategy Directorate

Lessons Learned Key lessons learnt from Pensim 2 development: Ø Tried to make the model too complicated – particularly the analytical side Ø We should have developed a set of simple modules for a Phase 1 delivery, then added complexity to the more important modules for Phase 2 Ø The structure of some of our modules was more complicated than the data would support, e. g. Labour Market process Ø Insufficient documentation produced by the Analytical team during the development (e. g. Base Data Creation) Ø Genesis Engine code is difficult to understand hence modify, although it is robust and does not require much modification Ø Some Genesis Engine functionality was over-ambitious and subsequently dropped (e. g. Alignment of polychotomous variables) 31 Model Development Unit Strategy Directorate

Any questions? 32 Model Development Unit Strategy Directorate