Postgraduate Course 6 Evidence based management What is

Postgraduate Course 6. Evidence based management: What is the best available evidence?

5 -step approach Postgraduate Course EBMgt is a 5 -step approach 1. Formulate an answerable question (PICOC) 2. Search for the best available evidence 3. Critically appraise the quality of the found evidence 4. Integrate the evidence with managerial expertise and organizational concerns and apply 5. Monitor and evaluate the results

Intermezzo Postgraduate Course Why are disciplines as such as psychoanalysis, astrology and parapsychology widely regarded as pseudo-science?

Falsifiability Postgraduate Course “It is easy to obtain evidence in favor of virtually any theory, but such ‘corroboration’ should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction, which might conceivably have been false. … A theory is scientific only if it is refutable by a conceivable event. Every genuine test of a scientific theory, then, is logically an attempt to refute or to falsify it. ” Carl Popper

Falsifiability Postgraduate Course “Inspect every piece of pseudoscience and you will find a security blanket, a thumb to suck, a skirt to hold. What have we to offer in exchange? Uncertainty! Insecurity!” Isaac Asimov

Research designs Postgraduate Course What is the BEST car?

Research designs Postgraduate Course Which design for which question?

Which design for which question? Postgraduate Course Explanation

Postgraduate Course Best research design?

Best available? Postgraduate Course

Postgraduate Course The best available evidence = Studies with the highest internal validity Studies with the highest external validity

Postgraduate Course 1. Best available evidence: internal validity

Internal validity Postgraduate Course internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible (confounding).

Causality Postgraduate Course When do we know there is causal relation? Three criteria: 1. the "cause" and the "effect" are related 2. the "cause" precedes the "effect" in time 3. there are no plausible alternative explanations for the observed effect

Causality Postgraduate Course Considerations for research: 1. Are the "cause" and the "effect” related: effect size INTERNAL VALIDITY 1. Does the "cause" precedes the "effect" in time: before and after measurement 2. Are there no plausible alternative explanations for the observed effect: randomization, control group

Internal validity Postgraduate Course internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible (confounding).

Methodological pitfalls Postgraduate Course § Bias § Confounding § Reverse Causation

Bias Postgraduate Course Bias: distortion of the outcome due to systematic errors caused by the way the study is designed or conducted. NB: If bias is not taken into account then any conclusions drawn may be wrong!

Forms of bias Postgraduate Course 1. Selection bias 2. Information (detection) bias 3. Performance bias 4. Exclusion (attrition) bias 5. Publication bias … … 30. …. .

Selection bias Postgraduate Course Error in the way participants in a study were selected. Because of this comparison groups differ in measured or unmeasured baseline characteristics.

Information bias Postgraduate Course Distortion of the outcome due to misinterpretation of information or systematic errors in the measurement of research variables which leads to misclassification. Information bias can be prevented by the use of standardized measurement instruments, hard outcome measures, validated questionnaires and objective, independent and blinded assessors. Types of information bias: § Reporting bias (recall bias) § Observer bias (interviewer bias, halo-effect)

Confounding Postgraduate Course Confounding is the idea that a 3 rd variable can distort or confuse (or confound. . ) a relationship between two other variables. For instance, when factor X causes disease Y, that relationship could be confounded by factor C that is associated with both factor X and disease Y. C would be an alternative explanation for the relationship observed between X and Y.

What are the confounders? Postgraduate Course 1. Shoe size & quality of handwriting 2. Body length & body weight 3. Number of storks & birth rate 4. Smoking youngsters & better lung function

Confounding Postgraduate Course

Postgraduate Course Correlation does not equal causation!

Postgraduate Course http: //kill-or-cure. heroku. com/

Reverse causation Postgraduate Course

Reverse causation Postgraduate Course ? Charismatic leaders Successful companies

Internal validity Postgraduate Course Cause and effect can be established only through the proper research design: no amount of statistical hand waving can turn correlations into conclusions about causation !!!

Levels of internal validity Postgraduate Course

Levels of internal validity Postgraduate Course It is shown that … It is likely that … There are signs that … Experts are of the opinion that …

Keep in mind! Postgraduate Course The levels of internal validity can only be used to determine which type of research is the best method to assess the validity of the cause-and-effect relationship that might exist between an intervention (or moderator) and its outcomes. In this respect, cross-sectional studies and case-studies have the ‘weakest’ design. This of course doesn’t mean that cross -sectional studies and case-studies have a weak design overall. After all, different types of research questions require different types of research designs. A case study for instance is clearly a strong design for assessing why or in which way an effect has occurred, but obviously not the most suitable design for assessing the strength of a possible cause-and-effect relationship.

Internal validity Postgraduate Course But … sometimes observational studies are as good as RCT’s When the size of effect is very large (swamps the bias)

Internal validity Postgraduate Course These treatments have not been tested in RCTs: are they supported by poor evidence? Heimlich manoeuvre Dehydration: drinking water Cardiac arrest: AED

Postgraduate Course 2. Best available evidence: external validity

External validity: generalizability Postgraduate Course Always ask yourself to what extent the evidence is generalizable to your situation: Ecological validity: Is your organization so different from those in the study that its results may be difficult to apply? Population validity: Is your population so different from those in the study that its results may be difficult to apply?

Generalizability Postgraduate Course Same Population? Same Intervention? Same Comparison? Same Outcome? Same Context?

Generalizability Postgraduate Course Keep in mind: What works in one narrowly defined setting might not work in another, but some psychological principles are generalizable to all human beings.

Internal vs external validity Postgraduate Course All research designs are flawed – though each is flawed differently. For instance, research designs with a high internal validity, such as controlled studies, may be less suited to generalization, which restricts their practical usability. Sample surveys and field research, on the other hand, have lower internal validity, but can sometimes be more useful for management practice. So there is always a trade off between internal validity (precision in control and measurements of variables) and external validity (generalizability with respect to populations, setting and context).

Best available evidence? Postgraduate Course internal validity external validity often high often low external validity sometimes high internal validity external validity often low often …?