Royal Military College of Canada Collge militaire royal

Royal Military College of Canada Collège militaire royal du Canada Dissent – Let’s Agree to Disagree Fred Cameron (Operational Analytics Canada) Jeff Appleget, Ph. D (US Naval Postgraduate School) Geoff Pond, PEng, Ph. D (Royal Military College of Canada) Contact: Fred. Cameron@Op. Analytics. ca

Wanted: A Methodical Way to “Organize” Dissent

Ranks and Rank Correlation • Two judges rank n objects • As a ranking each judge provides a permutation of the numbers 1. . . n • Kendall’s provides a correlation coefficient for a pair of judges • Kendall’s extension covers judges with ties • Kendall provides statistical tests of significance • A transformation of τ gives distance: (1 - τ)/2

Examples τ=1 d=0 τ = -1 d=1 τ = -0. 11 d = 0. 56

The Problem of m Rankings • Judges: m Objects: n • Each judge provides a permutation of 1. . . n (if no ties) • Extension allows a judge to submit ties: – 1 1 ► 2½ 2½ • Coefficient of Concordance: W W = 0. 828 F = (m-1) W / (1 -W), ν 1 = n – 1 – 2/m, ν 2 = (m-1) v 1 χr 2 = m (n-1) W, ν = n-1 Source of example: MG Kendall, Rank Correlation Methods, 4 th ed. , Griffin, London, 1970

Kendall’s Cautions “The provision of the above tests of significance [F test and 2 test of W] should not be allowed to obscure the desirability of examining the primary data to see if there any obvious effects present. ” “When a number of observers are suspected a priori to be heterogeneous in their tastes, it may obscure meaningful effects to assemble their rankings into a single group. ” Source: M. G. Kendall, Rank Correlation Methods, 4 th ed, Griffin, London, 1970, Sect 6. 15

Kendall’s “Extreme Case” For J 01 to J 20 W=0 Conclude: “No community of preference” But For J 01 to J 10, W = 1 For J 11 to J 20, W = 1 “The community of one set of observers has completely masked that of the other. ”

Alternatives for Group Ranking • • Rank sums or Borda count Condorcet voting Scores followed by weighted sums Analytic Hierarchy Process (AHP) • But we will use rank sums for the examples

Kendall’s Simple Example

Raw Ranks and Standardization

Analysis of Kendall’s Example

Cluster Analysis and Multidimensional Scaling Map borda R P d hclust (*, "average") -0. 04 -0. 020. 00 0. 02 0. 04 0. 06 0. 08 Q 0. 14 0. 10 0. 06 distance 0. 18 Cluster Dendrogram R borda Q P -0. 15 -0. 10 -0. 05 0. 00 0. 05 0. 10 0. 15

Second Example – Raw Ranks

Kendall’s W with p-value

Scaled Ranks with Group Ranking by Borda (and Reordered)

Pairwise Tau Coefficients

Pairwise Distances

Cluster Analysis and Multidimensional Scaling J 7 0. 2 0. 3 0. 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 1 J 3 J 2 -0. 1 0. 0 J 6 J 5 J 1 J 7 J 2 J 3 borda J 4 J 8 distance 0. 4 Cluster Dendrogram borda J 8 J 4 -0. 4 d hclust (*, "average") J 1 J 5 J 6 -0. 2 0. 0 0. 2 0. 4

The Four Agreeable Judges

Cluster Analysis and Multidimensional Scaling Cluster Dendrogram 0. 00 -0. 05 borda J 4 J 8 J 2 J 3 distance 0. 05 0. 10 0. 15 0. 05 J 2 J 4 J 8 borda J 3 -0. 15 -0. 10 -0. 05 0. 00 0. 05 0. 10 0. 15 d hclust (*, "average")

Four Other Judges

Cluster Analysis and Multidimensional Scaling d hclust (*, "average") -0. 10 -0. 05 0. 00 0. 05 0. 10 0. 15 borda J 6 J 5 J 1 J 7 0. 25 0. 30 0. 35 0. 40 distance Cluster Dendrogram J 1 J 5 J 7 -0. 3 J 6 borda -0. 2 -0. 1 0. 0 0. 1 0. 2

Conclusion Please TRY this at home!

R Script for Schools of Thought Analysis f <- file. choose() #get filename rr <- read. csv(f, row. names=1) #read file of judges' raw ranks sr <- apply(rr, MARGIN=2, rank) #scale to canonical form rs <- row. Sums(sr) #calculate rank sum borda <- rank(rs) #generate ranks for borda sr <- sr[, order(-cor(sr, borda, method="kendall"))] #reorder by agreement with borda srb <- cbind(sr, borda)[order(borda), ] #bind borda to judges' ranks w <- kendall(sr, correct=TRUE) #determine coeff concordance tau <- cor(srb, method="kendall") #calculate tau matrix d <- as. dist((1 -tau)/2) #calculate distances ca <- hclust(d, method="average") #determine cluster config mds <- cmdscale(d) #determine mds config x <- mds[, 1] #extract x coord for mds y <- mds[, 2] #extract y coord for mds plot(ca, ylab="distance") #plot hierarchical clusters plot (x, y, type="n", xlab="", ylab="", xlim=range(x)*1. 2, ylim=range(y)*1. 2) #set frame for mds map text(x, y, rownames(mds)) #add judge names to mds map

File of Raw Rankings , P, Q, R A, 1, 2, 2 B, 4, 1, 1 C, 2, 2, 4 D, 3, 4, 4 E, 4, 4, 4 F, 7, 8, 4 G, 6, 9, 8 H, 9, 6, 8 I, 7, 10, 8 J, 10, 6, 10 Note: 1. File is contains comma separated variables (CSV) with three judges and ten objects 2. File has labels in first row and first column 3. R will use only order: < > = 4. So, multiple 4’s will be treated as ties 5. No need to put ranks into canonical (standardized) form

Theory with Examples • Now in 3 rd edition, with about 100 additional pages • Chapter 17 on Multidimensional Scaling • Chapter 18 0 n Cluster Analysis

Royal Military College of Canada Collège militaire royal du Canada Questions and Answers (maybe) Also: Digressions, if time permits

Analytic Hierarchy Process – Two Choices • (1) Convert AHP scores to ranks and proceed as described • (2) Take each AHP score as a vector in nspace and use some definition of distance: – Pearson correlation or vector difference • Then proceed with cluster analysis and multidimensional scaling

The Alternatives Eight judges (J 1 to J 8) evaluated the following alternatives Source: The Technical Cooperation Program. Technology Requirements for Soldier Modernisation in the 2015 Timeframe JSA-AG 7 -2000 -01, TTCP, Washington, DC, 2000

AHP Results by Participant

Pairwise Pearson Correlation Coefficients

Cluster Analysis and Multidimensional Scaling Map 0. 2 0. 1 J 3 J 4 J 1 0. 0 0. 2 -0. 1 J 8 J 3 J 5 -0. 3 J 8 J 5 J 4 J 2 group J 7 d hclust (*, "average") J 7 J 2 J 6 group J 6 0. 1 0. 0 distance 0. 3 0. 4 Cluster Dendrogram -0. 4 -0. 2 0. 0 0. 2

R Script for AHP, part 1 f <- file. choose() #get filename ahp <- read. csv(f, row. names=1) #read file of ahp scores group <- ahp[ , length(ahp)] #get group scores only (last column) ahps <- ahp[ , -length(ahp)] #get participant ahp scores only ahps <- ahps[ , order(-cor(ahps, group))] #reorder by agreement with group ord <- cbind(ahps, group)[order(-group), ] #bind participants scores to group pcor <- cor(ord) #calculate pearson cor matrix d <- as. dist((1 -pcor)/2) #calculate distances ca <- hclust(d, method="average") #determine cluster config mds <- cmdscale(d) #determine mds config x <- mds[, 1] #extract x coord for mds y <- mds[, 2] #extract y coord for mds plot(ca, ylab="distance") #plot hierarchical clusters plot (x, y, type="n", xlab="", ylab="", xlim=range(x)*1. 2, ylim=range(y)*1. 2) #set frame for mds map text(x, y, rownames(mds)) #add judge names to mds map

R Script for AHP, part 2 #repeat Schools of Thought Analysis with ahp scores converted to rankings sr <- apply(-ahps, MARGIN=2, rank) #scale to canonical form rs <- row. Sums(sr) #calculate rank sum borda <- rank(rs) #generate ranks for borda sr <- sr[, order(-cor(sr, borda, method="kendall"))] #reorder by agreement with borda srb <- cbind(sr, borda)[order(borda), ] #bind borda to judges' ranks w <- kendall(sr, correct=TRUE) #determine coeff concordance tau <- cor(srb, method="kendall") #calculate tau matrix d <- as. dist((1 -tau)/2) #calculate distances ca <- hclust(d, method="average") #determine cluster config mds <- cmdscale(d) #determine mds config x <- mds[, 1] #extract x coord for mds y <- mds[, 2] #extract y coord for mds plot(ca, ylab="distance") #plot hierarchical clusters plot (x, y, type="n", xlab="", ylab="", xlim=range(x)*1. 2, ylim=range(y)*1. 2) #set frame for mds map text(x, y, rownames(mds)) #add judge names to mds map

Myths and Myth-busting • If we can agree on the facts, then we will agree on the decision – So, there should be no dissent • “If everyone is thinking alike, then somebody isn't thinking. ” General George S. Patton

Anscombe’s Quartet Set 1

Anscombe’s Quartet Set 2

Anscombe’s Quartet Set 3

Anscombe’s Quartet Set 4

We Need “Pictures”

Royal Military College of Canada Collège militaire royal du Canada Questions – Comments