VALIDITY AND RELIABILITY OF DIAGNOSIS PAGES 269 279

VALIDITY AND RELIABILITY OF DIAGNOSIS PAGES 269 -279

WHAT YOU WILL LEARN: • Reliability and validity of diagnosis • Relationship between validity of diagnosis • Reliability of the DSM • Validity of diagnosis • Research of validity of diagnosis: two approaches

RELIABILITY AND VALIDITY OF DIAGNOSIS: THE CONCEPTS • Diagnosis means “differentiating knowledge”, or telling something apart from everything else • Diagnosis is made on the basis of symptoms • Diagnosis in abnormal behavior means relating a pattern of behavior to a certain category • Two essential characteristics that define the quality of diagnosis are its reliability and validity • Diagnosis would be considered reliable if it is consistent across clinicians – referred to as inter-rater reliability. • There are two ways to establish inter-rater reliability of diagnosis • Audio/video-recording method: one clinician conducts the clinical interview and the interview gets recorded. This recording is then used by another clinician to arrive at a diagnosis independently. • Strength: same stimulus • Limitations: artificial, as the second clinician’s questions would have probably been slightly different as well as patients behavior and responses • Test-retest reliability: two clinicians conduct interviews with the same patient independently • Strength: overcomes the artificial issues of audio/video-recording method • Limitations: time. Patient’s symptoms may change naturally between interviews

RELIABILITY AND VALIDITY OF DIAGNOSIS: THE CONCEPTS • Reliability of diagnosis is quantifiable • Statistic that is most commonly used is the kappa coefficient • This statistic represents the amount of agreement that can be expected due to random chance • Kappa ranges from 0 to 1. (Copy table on page 270) • Validity of diagnosis relates to its accuracy – the degree to which a diagnostic system measures the behaviors that it intends to measure • A special form of validity is predictive validity – the ability to predict how the disorder will respond to treatment • Ex. If you are prescribed antidepressants and the drugs actually help, your diagnosis is said to have high predictive validity

RELATIONSHIP BETWEEN VALIDITY AND RELIABILITY OF DIAGNOSIS • To illustrate the ideas of validity and reliability, use the metaphor of shooting at a target (Figure 5. 5, page 271) • In diagnosis, you always want to hit the bull’s eye • However, this is hardly ever possible. • A reliable but invalid diagnosis is dangerous because the consensus between clinicians inspires trust in their judgement, but misdiagnosis will result in treatment issues. • NOTE: a valid measurement is reliable by definition, but a reliable measurement is not always valid.

RELIABILITY OF THE DSM • Research into earlier editions of the DSM showed its reliability to be alarmingly low • Beck et al (1962): found that agreement on specific diagnosis for 153 patients between two psychiatrists was only 54% • • • DSM-I was in use during this time Kendall (1974): studied almost 2, 000 patients who were admitted to a hospital from 1964 and then readmitted after 1969. • Found that schizophrenia was more often re-diagnosed as a form of depression than the other way around. • Study shows that diagnosis was not very consistent over time Focus shifted from explanation to observation in DSM-III • Overall, more reliable, but the reliability was not universal • Di Nardo et al (1993): found that using the DSM-III, there was “excellent” inter-rater reliability for such disorders as simple phobia and obsessivecompulsive disorders (OCD) but quite low reliability for generalized anxiety disorder (GAD). Reliability of DSM-III was further improved when the Structured Clinical Interview for DSM (SCID) was published. • A standardized set of questions. This facilitated consistency in the application of diagnostic criteria across clinicians – but the discrepancy between different disorders remained. • Williams et al (1992): using DSM-II and SCID conducted a test-retest reliability study of 592 patients • 0. 84 for bipolar disorder and substance abuse • 0. 64 for MDD • 0. 47 for social phobia Regier et al (2013): results of the DSM-5 field studies were “mixed”. • More than half of the diagnostic categories ranked in the top bands of “strong” or “moderate” reliability. On the other hand, some results such as those for MDD (a widely diagnosed category) were alarming: 0. 28

RELIABILITY OF THE DSM • Reliability of DSM-III was further improved when the Structured Clinical Interview for DSM (SCID) was published. • A standardized set of questions. This facilitated consistency in the application of diagnostic criteria across clinicians – but the discrepancy between different disorders remained. • Williams et al (1992): using DSM-II and SCID conducted a test-retest reliability study of 592 patients • 0. 84 for bipolar disorder and substance abuse • 0. 64 for MDD • 0. 47 for social phobia • Regier et al (2013): results of the DSM-5 field studies were “mixed”. • More than half of the diagnostic categories ranked in the top bands of “strong” or “moderate” reliability. On the other hand, some results such as those for MDD (a widely diagnosed category) were alarming: 0. 28

RESEARCH OF VALIDITY OF DIAGNOSIS: TWO APPROACHES • Validity of diagnosis, unlike reliability, cannot be quantified. • We can assess validity indirectly by intentionally attempting to discover lack of validity – looking for potential biases that compromise diagnosis. • There are two broad approaches to how research in this area may be organized: • Systematic biases in clinical judgment: compare two sets of diagnoses that should not differ if diagnosis is valid • Ex. Compare diagnosis (given to the same patients using the same clinical interview) of two groups of clinicians with different theoretical orientations. • Assessing the ability of psychiatrists to detect the disorder when the disorder is objectively known (but not disclosed to the psychiatrist) • Ex. The study of David Rosenhan (1973), “Being Sane in Insane Places”

ROSENHAN (1973) • Aim: to investigate if psychiatrists could tell the difference between sane and insane people • Methods: copy from pages 277 -278 • Results: copy from page 278 • Limitations: copy from page 279 • https: //www. youtube. com/watch? v=D 8 Oxd. GV_7 lo