Validity and Precision Julia Critchley Fiona Pearson Objectives

Validity and Precision Julia Critchley Fiona Pearson

Objectives By the end of the session you should have an understanding of: • the concepts of precision, accuracy, validity and reliability • how to maximise precision and validity in your study • ways to test the precision and validity of your study • how a lack of precision or accuracy may affect your study results

Good research • All epidemiological studies should aim to obtain the best information possible on parameters of interest amongst a study population • The study population should be as representative of the population of interest as possible • In order for this to be the case any error and/or bias must be avoided or reduced as much as is possible This will in turn: • • Avoid some types of bias Help to produce accurate, precise, valid and reliable measurements Achieve a high quality study Ensure the study draws true conclusions

Terminology Accuracy Precision How close a study parameter estimate (e. g. mean sbp) is to the true population value The extent to which a parameter estimate varies with repeated sampling. The smaller the variation Terminology the more precise the value Reliability The accuracy, consistency & stability of a test or study across differing situations Validity The likelihood that a study finding is true and free of bias

Accuracy, precision, validity & reliability: relationships Inter-related concepts of fundamental concern during study design Accuracy How good an estimate of population parameter Reliability Validity Truth of study findings How replicable are the results Precision How much do parameter estimates vary

Accuracy and Precision: Plausible circumstances True population parameter Less precise but still accurate Still precise but less accurate

Accuracy and Precision: Plausible circumstances Accurate and precise Inaccurate and precise Accurate but imprecise Inaccurate and imprecise

Validity Content validity If a questionnaire being used actually covers all aspects of the variable that is being studied. If the test is too narrow, then it will not predict what it claims. Face validity Whether or not the a survey has the right questions in order to answer the research questions that it aims to answer. Internal Validity When controlling all extraneous variables and the only variable influencing the results of a study is the one being manipulated by the researcher. External Validity The extent to which study results can be generalized or extended to others. Ecological Validity Degree to which behaviours observed and recorded reflect those occurring in natural settings and thus extent to which findings can be generalized.

Validity of categorical data • Concurrent or predictive validity most simply assessed in terms of tables (often 2 by 2 tables) of a diagnostic test or screening test for a disease • BUT – the principles apply equally to measurements of exposure

Instrument or Research Tool • “equipment hardware” – a red blood cell counter – a PH meter – an electronic weighing machine • “paperware” – a questionnaire – a weekly diet diary • “ peopleware” – observers/investigators – technicians

How good is the instrument or tool? • instrument • tool measurement – without bias or error – minimize bias • true value truth • measurement – valid/accurate – precise/reliable

IMPORTANT POINTS • precision depends on: – sample size – efficiency of the study • VIP influence on the power of a study • precision, reliability and consistency affected by RANDOM ERROR

ASSESSING PRECISION • Using S. D. s Variance (s 2 ( • Using Coefficient of variation = S. D. X • Using Kappa statistic • Using Cronbach's alpha

Testing reliability • Cronbach's alpha – measures internal consistency or reliability of a test instrument such as a validated questionnaire. The coefficient ranges in value from 0 to 1 the higher the score the more reliable the test

Random error is part of our experience that we cannot explain or predict. Precision and accuracy are always likely to be affected by random error To increase precision we can increase: the size of the study (total n) the efficiency of the study (e. g. selecting on rarer component disease or exposure) the proportion exposed or proportion of cases

Strategies for enhancing accuracy. 1 standardizing measurement methods • preparing study protocols • preparing operations manual • writing specific guidelines or instructions for making each measurement • serving as basis for describing methods when results are reported

ACCURACY DEFINITION: The degree to which the results of a measurement correspond to the true state or truth SYNONYM: • validity • conformity

IMPORTANT POINTS • accuracy is a function of “ SYSTEMATIC RROR” • VIP influence on the internal and external validity of the study • the greater the systematic error , the less accurate the variable

IMPORTANT POINTS • It is attributed to: – Methodological aspect of study design or analysis – Selection of subject – Quality of information obtained • Confounding • Effect Modification • Misclassification

ASSESSING ACCURACY Comparison with reference techniques Gold standards

Strategies for enhancing accuracy / precision . 1 Standardizing measurement methods. 2 Training and certifying the observers. 3 Refining the instruments. 4 Automating the instruments. 5 Making informal measures. 6 Blinding. 7 Calibrating the instrument

Strategies for enhancing accuracy • preparing operations manual – write down precisely: how to prepare environment and subject how to carry out and record interview how to calibrate instrument

Strategies for enhancing accuracy • writing specific guidelines or instructions for making the measurement uniform performance over the duration of study

Strategies for enhancing accuracy. 2 Training and certifying the observers • improving consistency of measurement techniques (several observers( • performing pilot study – to test the power of techniques specified in operations manual

Strategies for enhancing accuracy . 3 Refining the instruments • writing or spelling out questionnaires and interviews to increase clarity. 4 Automating the instruments • using automatic mechanical devices

Strategies for enhancing precision. 5 Repeating the measurement • impact of random error of any source can be reduced by – repeating measurement – using mean of the two or more readings

Evaluating the Validity of an Observational Study Does the study design seem the most appropriate for answering the research question? Does the sample size seem adequate (think about the commonality of the exposures and outcomes) and is data collection even across groups and covariates? Are the study groups comparable in all important aspects (list any for which they are not) except from their exposure status? Do the estimates appear precise? Is there potential bias due to differential loss to follow up? Is there potential recall or interviewer bias? Has confounding been considered and ruled out? Are results placed in the context of existing literature and are inconsistencies between studies suitably explained? Does the study have good external validity?

Why sample size may be too small True exposure status without a health risk Study assessed the exposure as: without a health risk with a health risk Statistical power = 1 - true negative: 1 - alpha error with a health risk beta error true positive: 1 -