VALIDITY AND RELIABILITY OF DIAGNOSIS PAGES 269 279

VALIDITY AND RELIABILITY OF DIAGNOSIS PAGES 269 -279

RELIABILITY AND VALIDITY OF DIAGNOSIS: THE CONCEPTS • Diagnosis means “differentiating knowledge”, or telling something apart from everything else • Diagnosis is made on the basis of symptoms • Diagnosis in abnormal behavior means relating a pattern of behavior to a certain category • Two essential characteristics that define the quality of diagnosis are its reliability and validity • Diagnosis would be considered reliable if it is consistent across clinicians – referred to as inter-rater reliability. • There are two ways to establish inter-rater reliability of diagnosis • Audio/video-recording method: one clinician conducts the clinical interview and the interview gets recorded. This recording is then used by another clinician to arrive at a diagnosis independently. • Strength: same stimulus • Limitations: artificial, as the second clinician’s questions would have probably been slightly different as well as patients behavior and responses • Test-retest reliability: two clinicians conduct interviews with the same patient independently • Strength: overcomes the artificial issues of audio/video-recording method • Limitations: time. Patient’s symptoms may change naturally between interviews

RELIABILITY AND VALIDITY OF DIAGNOSIS: THE CONCEPTS • Reliability of diagnosis is quantifiable • Statistic that is most commonly used is the kappa coefficient • This statistic represents the amount of agreement that can be expected due to random chance • Kappa ranges from 0 to 1 (closer to 1 is strong, closer to 0 is weak) • Validity of diagnosis relates to its accuracy – the degree to which a diagnostic system measures the behaviors that it intends to measure • A special form of validity is predictive validity – the ability to predict how the disorder will respond to treatment • Ex. If you are prescribed antidepressants and the drugs actually help, your diagnosis is said to have high predictive validity

RELATIONSHIP BETWEEN VALIDITY AND RELIABILITY OF DIAGNOSIS • To illustrate the ideas of validity and reliability, use the metaphor of shooting at a target (Figure 5. 5, page 271) • In diagnosis, you always want to hit the bull’s eye • However, this is hardly ever possible. • A reliable but invalid diagnosis is dangerous because the consensus between clinicians inspires trust in their judgement, but misdiagnosis will result in treatment issues. • NOTE: a valid measurement is reliable by definition, but a reliable measurement is not always valid.

RELIABILITY OF THE DSM • Research into earlier editions of the DSM showed its reliability to be alarmingly low • Beck et al (1962): found that agreement on specific diagnosis for 153 patients between two psychiatrists was only 54% • DSM-I was in use during this time • Kendall (1974): studied almost 2, 000 patients who were admitted to a hospital from 1964 and then readmitted after 1969. • Found that schizophrenia was more often re-diagnosed as a form of depression than the other way around. • Study shows that diagnosis was not very consistent over time • Focus shifted from explanation to observation in DSM-III • Overall, more reliable, but the reliability was not universal

RELIABILITY OF THE DSM • Reliability of DSM-III was further improved when the Structured Clinical Interview for DSM (SCID) was published. • A standardized set of questions. This facilitated consistency in the application of diagnostic criteria across clinicians – but the discrepancy between different disorders remained. • Williams et al (1992): using DSM-II and SCID conducted a test-retest reliability study of 592 patients • 0. 84 for bipolar disorder and substance abuse • 0. 64 for MDD • 0. 47 for social phobia • Regier et al (2013): results of the DSM-5 field studies were “mixed”. • More than half of the diagnostic categories ranked in the top bands of “strong” or “moderate” reliability. On the other hand, some results such as those for MDD (a widely diagnosed category) were alarming: 0. 28

RESEARCH OF VALIDITY OF DIAGNOSIS: TWO APPROACHES • Validity of diagnosis, unlike reliability, cannot be quantified. • We can assess validity indirectly by intentionally attempting to discover lack of validity – looking for potential biases that compromise diagnosis. • There are two broad approaches to how research in this area may be organized: • Systematic biases in clinical judgment: compare two sets of diagnoses that should not differ if diagnosis is valid • Ex. Compare diagnosis (given to the same patients using the same clinical interview) of two groups of clinicians with different theoretical orientations. • Assessing the ability of psychiatrists to detect the disorder when the disorder is objectively known (but not disclosed to the psychiatrist) • Ex. The study of David Rosenhan (1973), “Being Sane in Insane Places”

ROSENHAN (1973) • Aim: To investigate if psychiatrists could tell the difference between stable and unstable people • Methods: • Eight mentally healthy subjects (including experimenter Rosenhan) presented themselves as pseudo-patients seeking admittance. They did so at 12 hospitals across 5 different states. They called hospitals and scheduled an appointment • Complained of hearing voices that said “empty”, “hollow”, and “thud”. Described voice as unfamiliar, the same sex as them, and often unclear except for those 3 words • Other than this made up fact, they were instructed to be truthful when providing information about themselves and to act normally. • To prevent issues with future employment records, excluded their names and provided fake names • The “pseudo patients” were to get discharged from the hospital by their own means, convincing staff of their sanity on their own

ROSENHAN (1973) • Methods (cont): • Once admitted to the psychiatric ward, “pseudo patients” stopped displaying symptoms • Rosenhan noted that “pseudo patients” were nervous due to fear of being exposed and embarrassed. But, this nervousness quickly went away • The pseudo patients behaved normally, and when asked by staff answered that they felt fine and stopped experiencing symptoms. Hospital records even note that they were friendly and cooperative. • All instructions were followed, but medication was not swallowed • Participants discreetly noted their observations of the ward, patients and staff • Almost immediately after admission, participants sought to be discharged from the hospital

• Results: ROSENHAN (1973) • All but one pseudo-patient were admitted into the hospital based on the diagnosis of “schizophrenia”. Each was discharged with a diagnosis of “schizophrenia in remission” • Length of hospital stay varied between 7 and 52 days (average stay was 19 days) • Not one “pseudo patient” was detected by any hospital staff (although other patients were more observant and speculated that they weren’t real patients) • Staff misinterpreted the “pseudo patients” normal behavior as symptoms of their illness • Also, participants noted that after admission they were largely ignored by staff during their stay • Conclusion: Psychiatrists could not reliably distinguish normal individuals from the mentally ill https: //www. youtube. com/watch? v=D 8 Oxd. GV_7 lo