Handbook on HWF planning methodologies MODULE 5 Evaluation

Handbook on HWF planning methodologies MODULE 5 Evaluation of the selected practices Annalisa Malgieri & Paolo Michelutti WP 5 team _______________ Turin, 18 – 19 September 2014

7 Consolidated planning experiences as starting point…solid foundations for the selection Quantitative projections Both supply and demand European models

The first step to the evaluation…the checklist 5 SECTIONS 32 ITEMS 3 OPTIONS FOR THE ANSWERS SHARED AND VALIDATED BY PARTNERS IN JULY 2014 THOUGHT AS A «SELF-EVALUATION» TOOL 7 COMLETED CHECK LISTS RECEIVED HARMONIZATION OF THE EVALUATION

Before starting some important clarifications Ø The check list has not the ambition to be exhaustive of all the indicators useful for the evaluation of a planning system Ø The purpose of the check list is not to drawn up a classification of the goodness of different aspects of the planning systems in the analyzed Countries, but to furnish the most objective and standardized interpretation for the individuation of the good practices Ø Related evaluations are only referred to the current situation and to the practices today in use. This means that we haven’t taken into account practices no more in use and practices not still consolidated

How to use the check list? The check list allows for different kinds of evaluations: Ø an evaluation and identification of strengths and weaknesses common to the different planning systems analyzed Ø an evaluation and identification of the specific good practices within each planning system

2 dimensions to explore for each item … IMPORTANCE Each item is identified by coordinates (importance, performance) ITEM E. 6 ………. . ITEM B. 1 ITEM A. 3 ITEM A. 2 PERFORMANCE

4 different areas identified MAP OF STRENGHTS AND WEAKNESSES

Let’s take a look at the results…(1/4) AREA OF EXCELLENCE B. 1 The assumptions in the forecasting model are regularly re-assessed in light of changing circumstances, new data, new policies and programs. C. 1 Data used in the forecasting model are regularly updated in order to have a realistic picture of what is the current situation of the HWF demand supply. C. 2 The HWF planning tool considers the entire work life cycle (supply side). D. 1 There is a control and continuous monitoring process to check if the planning objectives are reached. D. 6 The planning system allows the Ministry of Education to fix the number of positions for trainees E. 3 The parameters in the forecasting model are discussed and agree with stakeholders. E. 5 Tasks and corresponding competences of people and institutions involved in the planning process are clearly defined.

Let’s take a look at the results…(2/4) AREA OF IMPROVEMENT A. 1 The goals of the HWF planning system are explicit and clearly defined. A. 4 The HWF planning system includes short- and long-term targets with multiple time frames. B. 2 B. 3 B. 5 B. 9 B. 10 B. 11 C. 3 D. 2 D. 6 E. 4 The HWF forecasting model is flexible and can easily be changed and adapted by authorities competent for planning. The HWF planning tool estimates any current imbalances and it doesn’t assume a current equilibrium between supply and demand. The HWF forecasting model produces different scenarios on the base of different patterns values (i. e. university training capacity, attrition rates, retirement patterns, migration flows). The HWF planning system takes into account the interaction between professions using a multi-professional teamwork approach. The planning system forecasts the trend of dependence of the health care system on foreign trained HWF personnel. The HWF planning system takes into account budget constraints in its scenarios and it allows to evaluate alternative allocations of resources (i. e. budget for specialist training for medical doctors). Data collected on current stock of HWF allow to know: number of active professionals; number of full-time equivalent; types of providers; where they work; their skills; the services they provide; workloads. The planning system develops tools (i. e. check lists, guidelines) to evaluate and inform the decision making process on its own planning capacity (self-evaluation). The HWF planning system defines objectives and actions to address geographical mal-distribution and imbalances. Stakeholders participate in the elaboration of scenarios.

Let’s take a look at the results…(3/4) AREA OF MAINTENANCE B. 8 The forecasting model can be applied to different health professions. C. 4 The HWF planning tool uses demand data which allow estimating the health care utilization and the population needs, in addition to size and structure (age and sex) of the population. E. 6 There is a formal mechanisms for health education planners to discuss on HWF needs

Let’s take a look at the results…(4/4) AREA OF INDIFFERENCE A. 2 A. 3 B. 4 B. 6 B. 7 D. 3 D. 5 D. 7 E. 1 E. 2 The goals of the HWF planning system are based on a comprehensive and updated analysis of the future health needs of the population. The goals of the HWF planning are based on the analysis of skills needed to deliver planned health services in the future. The HWF planning tool considers the expected impacts of health service delivery changes. The HWF projections are specific to the different healthcare sectors (e. g. Acute Care vs. Primare Care; Public sector vs. Private Sector). The HWF planning is harmonized with social and economic development and the planning of other related sectors: i. e. basic assistance, social protection, prevention, sport, education. The planning system allows policy makers to receive updated reports on demand in a reasonable timeframe (3 to 6 months). The HWF planning system defines objectives and actions on retirement patterns and attrition rates. The planning system provides indicators that regularly measure the impact on the health care system in terms of improving, attraction and keeping skilled HWF (professionals are satisfied with their work, working environment and career paths). The HWF planning system provides guidelines on how to communicate the results to policy-makers and other stakeholders. The IT forecasting tool has a “user friendly” interface which allows stakeholders an easy interaction.

1 2 3 IMPORTANCE On the road for the good practices’ selection FOCUS ON PERFORMANCE DIMENSION PERFORMANCE Not just High (+) and Low (-) level but an exact value of performance for each item (score) Synthesis and analysis of results: Per item model Per section Per planning

Synoptic table of performance SECTION BELGIUM DENMARK ENGLAND FINLAND NETHERLANDS NORWAY SPAIN A - GOALS 25% 100% 50% 75% 38% B - FORECASTING 50% 45% 82% 50% 68% 32% 41% C - DATA 100% 63% 100% 88% 75% D - LINK TO POLICY ACTIONS 71% 36% 93% 43% 57% 50% 36% E - ORGANISATION 75% 58% 75% 67% 33% 50% PAY ATTENTION: IT’S NOT A RANKING LIST BUT JUST A GUIDE TO HELP THE SELECTION OF GOOD PRACTICES FOR EACH SPECIFIC ASPECT OF A HWF PLANNING SYSTEM

The selected good practices BELGIUM Good practice on involvement of stakeholders, stakeholders elaboration of a common proposal, reporting and advising the policy makers (and how it's important to have the right staff to manage this process). With examples of reports to the policy makers. Good practice on data collection of the supply side, with an in-depth analysis on how side the process works, how data on "practicing" and "active" are collected, how FTE is calculated. Good practice in monitoring the applied planning and enabling policy discussion. DENMARK Good practice on the supply side forecasting "easy and flexible", main aspects of the flexible" model, available tools, etc.

The selected good practices ENGLAND Good practice on to define explicit, transparent and specific goals on HWF planning Good practice on stakeholder involvement: the experiences of the "call for evidence" and the process of "triangulation” evidence" Good practice on data collection process FINLAND Good practice on HWF planning harmonized with social and economic development and planning of other related sectors

The selected good practices NETHERLANDS Good practice on how to assess the current imbalances: a concrete example (maybe imbalances on general practitioners) of which indicators and standard used to estimate current imbalances Good practice on forecasting the demand side: in particular for dentist side Good practice on data collection, in particular on FTE estimation (examples, detailed collection description of the process, tools, etc). Good practice on self-evaluation of the effectiveness of the planning model (monitoring process and consumer satisfaction survey): examples and tools.

The selected good practices NORWAY Good practice on forecasting the HWF demand SPAIN Good practices on multi regional planning: organization aspects and forecasting model planning with special attention on in-land migration impacts. Good practice on how HWF planning system takes into account budget constraints in its constraints scenarios and it allows to evaluate alternative allocation of resources.

Methodology for Strengths/Weaknesses MAP (1/2) PERFORMANCE DIMENSION EVALUATION Not at all (Na. A) Partially (P) Completely (C) ASSOCIATED POINTS 0 0, 5 1 1 Translation of the evaluation given in number values (see table) 2 Sum of the score obtained by each item in all 7 planning models (absolute score) 3 Relative score calculation (absolute score/max score in percentage) i. e. ITEM B. 1 4 ITEM_NUMBER ITEM_DESCRIPTION BELGIUM B. 1 The assumptions in the forecasting model are regularly re-assessed in light of changing circumstances, new data, new policies and programs. C DENMAR NETHERL K ENGLAND FINLAND ANDS NORWAY SPAIN C C P C Score 5, 5 Max_Score Rel_Score 6 92% Items with relative score > 75% have been defined with high performance (+) Items with relative score <= 75% have been defined with low performance (-)

Methodology for Strengths/Weaknesses MAP (2/2) IMPORTANCE DIMENSION PRINCIPLE: inclusion/not inclusion in a “minimum” health workforce planning model As per “minimum” model it is considered a “start-up” model containing the elements necessary for an efficient human resources planning, with particular attention to its use in those Countries/Regions that for the first time approach to theme of health workforce planning in quantitative terms CRITERIA: Items considered essential to be included in a minimum model have been evaluated with high importance (+) Items considered not to be essential to be included in a minimum model have been evaluated with low importance (-)

Methodology for good practices’ selection (1/2) PURE MATHEMATICAL CRITERIA EVALUATION Not at all (Na. A) Partially (P) Completely (C) ASSOCIATED POINTS 0 0, 5 1 1 Translation of the evaluation given in number values (see table) 2 Sum of the score obtained by the item for each section and for each model 3 Relative score calculation (absolute score/max score in percentage) i. e. BELGIUM SECTION A B C D E 4 Na. A 2 4 0 1 1 P 2 3 0 2 1 C 0 4 4 Score 1 5, 5 4, 5 Max_Score Rel_Score 4 25% 11 50% 4 100% 7 71% 6 75% RANK 5 4 1 3 2 Comparison of the results among system planning models analyzed for each section to identify the good practice

Methodology for good practices’ selection (1/2) ADDITIONAL CRITERIA 1 2 Good performance on items placed in “improvement area” Good performance on items considered very important based on Milan and Florence workshop evidences ü i. e. budget constraints ü i. e. FTE calculation 3 Good performance on specific aspect of the planning system which are useful for the implementation and the improvement 4 Unique experience on a specific and interesting aspect

Conclusion THE OBJECTIVE OF THIS WORK IS NOT TO MAKE A RANKING OF THE ANALYZED HWF PLANNING SYSTEMS THEY ALREADY REPRESENT GOOD EXPERIENCES ON HWF PLANNING THE METHODOLOGY APPLIED HAS BEEN STUDIED IN ORDER TO: - identify common strengths and weaknesses - develop a tool in order to define a prior area of intervention (improvement) - take into account all aspects to be included in a minimum planning model (implementation) THE FINAL SCOPE IS TAKING ADVANTAGE FROM THE EXPERIENCE OF SOMEONE TO GROW UP ALL TOGETHER, THANKS TO THE EXCHANGE OF THE GOOD PRACTICES