GRADE approach to assess certainty of evidence in

GRADE approach to assess certainty of evidence in HTA / Użycie podejścia GRADE do oceny pewności dowodów w HTA Małgorzata Bała, MD, Ph. D Chair of Epidemiology and Preventive Medicine Department of Hygiene and Dietetics 1

Uncertainty 2

Bradford Hill criteria Association or causation Strength (effect size and precision) Consistency (replication) Specificity (PICO) Temporality (longitudinal) Biological plausibility Biological gradient (dose-response) Coherence (laboratory and epidemiological data) Experimental evidence (randomized trial data) Analogy (existing similar exposure-risk associations) The Environment and Disease: Association or Causation? Proc Royal Soc Med 1965; 58: 295 -300. 3

GRADE certainty (quality) of evidence US GRADE network 4

Certainty (quality) of evidence Ratings Definitions We are very confident that the true effect lies close to High certainty that of the estimate of the effect Moderate certainty Low certainty Very low certainty GRADE Handbook We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect 5

Quality of evidence GRADE Handbook 6

Cochrane review Antibiotics in acute otitis media Venekamp et al. , Cochrane Database of Systematic Review 2004 7

Evidence to decision framework (Et. D) 4 GRADE Working Group 4 DECIDE Project (Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence) (2011 -2015) 4 iterative process, brainstorming workshops, consultation with an advisory group, user testing, feedback, application to different types of decisions, including coverage decisions Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 8

Evidence to decision framework 4 General structure Question Assessment (12 criteria) Conclusions 4 Criteria – similar to guidelines from a population perspective 4 Systematic and transparent approach for decision making Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 9

Evidence to decision framework 4 Explicit criteria – for each research evidence, additional considerations, judgements 4 Assessment criteria: Priority of the problem 8 Desirable, undesirable effects 8 Certainty of evidence 8 Values 8 Balance of effects 8 Resources required, certainty of evidence of required resources 8 Equity, resource use, cost-effectiveness, acceptability, feasibility 8 Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 10

Evidence to Decision (Et. D) conceptual map workflow Moberg et al. Health Research Policy and Systems (2018) 16: 45 11

Et. D - question Patient Intervention Comparison Main outcome Setting Perspective Subgroups Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 12

Example https: //ietd. epistemonikos. org/#/frameworks/54 eb 9 ff 02 b 38677806 a d 849 b/question_details Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; 13

Et. D for coverage decisions Is the problem a priority • Are the consequences of the problem serious (that is, severe or important in terms of the potential benefits or savings)? • Is it a recognised priority (such as based on a political or policy decision)? Don’t know Varies No Probably Yes Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 14

Et. D for coverage decisions How substantial are the desirable anticipated effects? How substantial are the undesirable anticipated effects? What is the overall certainty of the evidence of effects? Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 15

ETD SOF https: //isof. epistemonikos. org/#/finding/54 d 4 f 9 c 3 f 30 d 0 c 1 d 86385 ccc Evidence profile https: //ietd. epistemonikos. org/#/frameworks/54 eb 9 ff 02 b 38677806 ad 849 b/evidence_profile 16

Et. D for coverage decisions How substantial are the desirable anticipated effects? Don’t know Varies Trivial Small Moderate Large How substantial are the undesirable anticipated effects? Don’t know Varies Large Moderate Small Trivial What is the overall certainty of the evidence of effects? High Moderate Low Very low Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 17

Et. D for coverage decisions Is there important uncertainty about or variability in how much people value the main outcomes? Is there important uncertainty about how much people value each of the main outcomes? Is there important variability about how much people value each of the main outcomes? Important uncertainty Possibly important Possibly no uncertainty important uncertainty No important uncertainty Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 18

Et. D for coverage decisions Does the balance between desirable and undesirable effects favour the intervention or the comparison? • Judgments regarding each of the four preceding criteria (desirable, undesirable, certainty of evidence, values) • To what extent do the following considerations influence the balance between the desirable and undesirable effects:  - How much less people value outcomes that are in the future compared to outcomes that occur now (their discount rates)?  - People’s attitudes towards undesirable effects (how risk averse they are)?  - People’s attitudes towards desirable effects (how risk seeking they are)? Don’t know Varies Favours the comparison Probably favours the comparison Does not Probably Favours favours the either the intervention interventio tion n or the Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), comparison 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 19

Et. D for coverage decisions How large are the resource requirements (costs)? How large is the difference in each item of resource use for which fewer resources are required? How large is the difference in each item of resource use for which more resources are required? https: //ietd. epistemonikos. org/#/frameworks/54 eb 9 ff 02 b 38677806 ad 849 b/ criteria/resources_required Don’t know Varies Large costs Moderate costs Negligible costs or savings Moder Large ate saving s s Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 20

Et. D for coverage decisions What is the certainty of the evidence of resource requirements (costs)? • Have all-important items of resource use that may differ between the options being considered been identified? • How certain is the evidence of differences in resource use between the options being considered? • How certain is the cost of the items of resource use that differ between the options being considered? • Is there important variability in the cost of the items of resource use that differ between the options being considered? High Moderate Low Very low Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 21

Et. D for coverage decisions Does the cost effectiveness of the option favour the option or the comparison? • Judgments regarding each of the six preceding criteria • Is the cost effectiveness ratio sensitive to one-way sensitivity analyses? • Is the cost effectiveness ratio sensitive to multivariable sensitivity analysis? • Is the economic evaluation on which the cost effectiveness estimate is based reliable? • Is the economic evaluation on which the cost effectiveness estimate is based applicable to the setting(s) of interest? Don’t know Varies Favours the comparison Probably favours the comparison Does not favour either the interventio n or the comparison Probably favours the intervention Favours the interven tion Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 22

Et. D for coverage decisions What would be the impact on health equity? Are there groups or settings that might be disadvantaged in relation to the problem or interventions (options) that are considered? • Are there plausible reasons for anticipating differences in the relative effectiveness of the intervention (option) for disadvantaged groups or settings? • Are there different baseline conditions across groups or settings that affect the absolute effectiveness of the intervention or the importance of the problem for disadvantaged groups or settings? • Are there important considerations that should be made when implementing the intervention (option) in order to ensure that inequities are reduced, if possible, and that they are not increased? Don’t know Varies Reduced Probably no reduced impact Probably increased Increased Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 23

Et. D for coverage decisions Is the option acceptable to key stakeholders? • Are there key stakeholders who would not accept the distribution of the benefits, harms and costs? • Are there key stakeholders who would not accept the costs or undesirable effects in the short term for desirable effects (benefits) in the future? • Are there key stakeholders who would not agree with the importance (value) attached to the desirable or undesirable effects (because of how they might be affected personally or because of their perceptions of the relative importance of the effects for others)? • Would the intervention adversely affect people’s autonomy? • Are there key stakeholders who would disapprove of the intervention morally, for reasons other than its effects on people’s autonomy (such as in regard to ethical principles such as no maleficence, beneficence, or justice)? Don’t know Yes Probably No No Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 24

Et. D for coverage decisions Is the option feasible to implement? For coverage decisions:   • Is coverage of the intervention sustainable?   • Is it feasible to ensure appropriate use for approved indications?   • Is inappropriate use (indications that are not approved) an important concern?   • Is access to the intervention an important concern?   • Are there important legal or bureaucratic or legal constraints that make it difficult or impossible to cover the intervention? Don’t know Varies No Probably Yes Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 Yes 25

Conclusion 4 Summary of all judgments 4 Type of decision Do not cover Cover with evidence development Cover with price negotiations Restricted coverage Cover Stop covering opportunistic screening for asymptomatic men, keep on covering just for the ones with familial history of prostate cancer. Justification: No evidence of efficacy on mortality in general male population >50 years. Minor and major adverse events such as bleeding, bruising, short term anxiety, overdiagnosis and overtreatment, erectile dysfunction and incontinence, infections, blood loss requiring transfusion, pneumonia. Parmelli et al. Intl J Technology Assessment in Health Care, 33: 2 (2017), 176– 182; Alonso-Coello P. BMJ 2016; 353: i 2016 26

Evidence to decision in coverage decisions Systematic and transparent approach Ensuring systematic consideration of key criteria that determine the decision Use of the best available evidence 27

GINAHTA 28

4 Thank you 29