Cognitive Systems Engineering Argumentation Theory and the DataFrame

Cognitive Systems Engineering, Argumentation Theory and the Data-Frame Model: A theoretical framework for supporting the Visual Analytics cognitive work domain 31 July 2012 Professor William Wong and Dr Simon Attfield Interaction Design Center Middlesex University, London, UK

UKVAC Joint Research Project • UKVAC – A consortium of five UK universities: – Middlesex (lead), Imperial, UCL, Oxford, Bangor. • UKVAC JRP – Funded by DHS and UKGOV. • Two proxy intel. problems: – Noble Laureates: Who will be the short-listed candidates for the Nobel Laureates in 20 XX? – Flight data: Questions about 120 million records of all US commercial flights from October 1987 to April 2008. e. g. When is the best time to fly? Can you detect cascading delays? • WP 5 (Middlesex) - Design prototype for visually representing reasoning to support sensemaking. • What role does the visual representation of reasoning have in sensemaking?

Design Challenges • Exploration under Uncertainty – Supporting search and sense-making • Revealing the nature of the data – what’s in it? – Through a keyhole • What is the structure of the data set? – How are they organised? • Relationships within and between the data – Meaningful only within the context of the goals • Constructing new assemblies of evidence – Explain, elaborate, question, reframe • Tracing analytical reasoning: Conclusion Pathways • Supporting work environment: Reasoning Workspace 3

Information seeking uncertainty • Taylor’s work on question negotiation at the library reference desk (1960 s). – Can be little more than a vague sense of something missing. – A query is "a description of an area of doubt in which the question is open ended, negotiable and dynamic”. • Belkin and Brooks ASK hypothesis (1980 s) – Information seekers are faced with an anomaly and are, in general, unable to specify precisely what they need to address it. • Kulthau (1990 s), Vakkari (2000 s) – Uncertainty gives way to focus formulation. – During uncertain stages needs are vague and information system interaction is most difficult.

Uncertainty and Sensemaking • Info. needs evolve and change through interaction with information and the development of higherorder reasoning. • Uncertainty and sensemaking are inversely related. Model of Sensemaking in intel. (Pirolli and Card , 2005) Model of Sensemaking in e-discovery (Attfield and Blandford, 2011)

Exploratory dialogue • Look-up based IR model supplemented with concepts from Interactive Information Retrieval (IIR). • Cue exploratory search systems (ESS) which deploy mechanisms for closely coupled interaction which recognise the importance of an exploratory dialogue: – Relevance feedback; – Faceted search; – Visualisation; • Example – At TRe. C Brassil and Hogan demonstrated very high recall and precision using a form of sociotechnical relevance feedback.

Big Data and the Cost Structure of Exploration • Big Data can increase the user-cost of exploration. • Information systems (VA or otherwise) don’t support the user in representing the reasoning process. • Its not just about information! Data Frame Model of Sensemaking. (Klein et al. 2007) • Where cost of exploration is higher, the payoff for representing higher-order reasoning is likely to be more favourable.

Toulmin’s Argument Model 8 http: //www. lawaugh. com/maps/

Argument mapping to guide VA interaction Wigmore diagrams Toulmin argument maps CSE – Abstraction Hierarchy Data/Frame model Visual Language for expressing intelligence assessments Intelligence assessments Academic Literature Interviews • Easily creating and modifying assessments; • Sharing and collaborative critiquing; • Applying analytic rigor and reducing bias; • Dealing with and encoding uncertainty and confidence; • Targeting resources towards profitable/critical parts of an analysis.

Anticipated Benefits • Targeting resources towards profitable/critical parts of analysis. • Easily creating and modifying assessments; • Sharing and collaborative critiquing; • Applying analytic rigor and reducing bias; • Dealing with and encoding uncertainty and confidence;

Design of Visualisations • what the design means is usually obvious to the designer, but less so to the viewer 11

12 http: //radar. oreilly. com/2011/11/visualization-popular-science-archive. html

Information Analysts: What do they do? CTA-based Group Discussion • Is it just Information search and retrieval? 13

Information Analysts: What do they do? CTA-based Group Discussion (n=20 intel analyst) • • Is it just Information search and retrieval? Ill-formed queries and exploratory searches Learning about the topic that they are searching Connecting (context, goals, background) with the data – “Oh, is that what it means …!” • Filter, classify, index and organize the data, or carry out some form of analysis on the data • Attempt to make sense of the results or of the data from the way the data is organized – Structures and relationships within the data • This sense-making process involves constructing explanations from the data: this is the process of connecting with the data, and then developing and testing the explanations derived from the data and the analysis of the data. 14

Learning from Cognitive Systems Engineering Vicente, K. J. (1999). Cognitive Work Analysis: Toward safe, productive, and healthy computer-based work. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. , Publishers. Rasmussen, J. , Pejtersen, A. M. , & Goodstein, L. P. (1994). Cognitive systems engineering. New York: Wiley. Vicente, K. J. , & Rasmussen, J. (1990). The Ecology of Human Machine-Systems II: Mediating "direct perception" in complex work domains. Ecological Psychology, 2(3), 207 -249. • Cognitive Systems Engineering – Cognitive Work Analysis, Means-Ends Analysis, Abstraction Hierarchy, Decision Ladder, Control Task Analysis, … – Identifying structures – Identifying functional (meaningful) relationships – In relation to goals and how experts reason about the problem • In Visual Analytics - Helping data tell their story! • Enabling a user to create a representation of the inferences they draw about their domain 15

Mimic Diagram 16 Jamieson, G. A. , & Vicente, K. J. (2001). Ecological interface design for petrochemical applications: supporting operator adaptation, continuous learning, and distributed, cognitive work. Computers and Chemical Engineering, 25, 1055 - 1975.

Ecological Interface Design 17 Jamieson, G. A. , & Vicente, K. J. (2001). Ecological interface design for petrochemical applications: supporting operator adaptation, continuous learning, and distributed, cognitive work. Computers and Chemical Engineering, 25, 1055 - 1975.

Ecological Interface Design of DURESS II Vicente, K. J. & Rasmussen, J. (1990). The ecology of human-machine systems II: Mediating "direct perception" in complex work domains. Ecological Psychology, 2, 207 -249. Vicente, K. J. & Rasmussen, J. (1992). Ecological Interface Design: Theoretical foundations. IEEE Transactions on Systems, Man and Cybernetics, 22, 589 -606. 18

Time Tunnel Display: Making functional relationships visible 19 Bennett, K. B. , Payne, M. , & Walters, B. (2005). An evaluation of a "Time Tunnel" display format for the presentation of temporal information. Human Factors, 47(2), 342 -359.

Functional Relationships in Information Space • How to structure the relationships that underlie data, evidence that support our descriptions of what is observed? – Cognitive Systems Engineering • Cognitive Work Analysis (Vicente, 1999; Rasmussen) • Functional Relationships, affordances to support thinking – Representation Design • E. g. Avoid strong visual geometries – Argumentation Theory • Toulmin (1958) The Uses of Argument. Cambridge, England: Cambridge University Press. 20

Data-Frame Model of Sense-making 21 Klein, G. , Philips, J. K. , Rall, E. L. , & Peluso, D. A. (2007). A data-frame theory of sense-making. In R. R. Hoffman (Ed. ), Expertise Out of Context: Proceedings of the Sixth International Conference on Naturalistic Decision Making (pp. 113 -155). New York: Lawrence Erlbaum Associates. Klein, G. , Moon, B. , & Hoffman, R. R. (2006). Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent Systems, 21(5), 88 -92.

Interactive Visualization Principled Approach • Information Design Principles – – – Focus+Context Proximity-Compatibility Principle Gestalt Principles of Form Perception Principle of Visual Affordance Ecological Interface Design Representation of Functional Relationships • Cognitive Systems Engineering • Argumentation Theory, e. g. Toulmin’s Diagrams • Essential Foundation: Tightly coupled perception-action cycle to develop and maintain cognitive momentum – – – Neisser, U. (1976). Cognition and Reality. New York, NY: W. H. Freeman and Company. Arnheim, R. (1969). Visual Thinking. Berkeley and Los Angeles, CA: University of California Press. Mc. Kim, R. H. (1980). Experiences in Visual Thinking. Belmont, CA: Brooks/Cole Publishing Company. • Prompt the asking of Questions 22

The 20 Representation Design Problems Wong, B. L. W. , & Varga, M. (2012). Blackholes, keyholes and brown worms: challenges in sense making Proceedings of HFES 2012, the 56 th Annual Meeting of the Human Factors and Ergonomics Society, Boston, MA, 22 -26 October, 2012 (pp. Accepted for publication). Santa Monica, CA: HFES Press. Data Space – what’s available? changed? What’s there? • – – – Seeing a large data set and reasoning space through a small keyhole (the Keyhole problem). Handling missing data (the Black Hole problem) Handling deceptive / misleading data (the Brown Work problem) Handling contradictory data. Recognising and representing anomalous changes. Developing a sense of what is in the data – exploring what is there. Analysis Space – what behaviours, relationships, patterns? Aggregating and reconciling multiple points of view or predictions. • • – – – – Handling strength of evidence (including subjective and objective measures of strength) + contribution of different pieces of evidence to a conclusion. Handling uncertainty in data and information. Predicting and representing emergent behaviour. Predicting the path of cascading failures or effects. Identifying and representing trends. Finding the needle in the haystack (or knowing what is chaff – i. e. info of no or low value) Annotating, remembering, re-visiting, setting aside. Provenance and tracing analytic reasoning. Hypothesis Space – collate, marshal and test evidence. • – – – Collate evidence for evidential reasoning. Collate and represent evidence over time and space. Represent static and dynamic relationships between the data. General Issues • 23 – – Scalability and reusability. Seamless integration of workflow between data space, analytic space and hypothesis spaces.

(1) The Keyhole Problem Visually supported analytic reasoning By nature Inter-disciplinary Lack of the ‘big picture’ Jig-saw puzzle (not one, but many) Large, multi-sourced, stream, changing data sets Frame of Reference Keyhole problem

(2) The Black Hole Problem: Missing Data • Incomplete datasets – how do we identify, visualize and alert the user that there are or may be missing data? xxxxxxxxxxxxxxxxx Sequential but not temporal (aggregation) xxxxxxxxxxxx x Sequential and temporal <= showing the blackholes xxxxxxxx xxxx x Deliberate occurrence? <= blackholes encourage questions to be asked x x x x x x x x xx x xx 25 Automatic averaging / substitution => are there missing data?

(3) Brown Worm Problem: Deceptive and Misleading Data • Data is plausible and intended to mislead • Strength of evidence display and Alternate plausibilities – One approach based on cause and effect and human memory limitations (capacity, and propensity for misremembering) – Visually persistent data showing contributions to conclusions in ways that permit direct and rapid visual comparisons – Questioning a frame; Re-framing and re-connecting 26

27

Grounds (1) 28 Claim

Grounds (1) 29 Warrant Rules for interpreting Validity? Authenticity? Claim

Backing (1) Backing 1 Grounds (1) Backing (1) 30 Warrant Validity? Authenticity? Backing (1) Claim

Backing (1) Backing 1 Qualifiers Backing 1 Grounds (1) Backing (1) 31 Warrant Validity? Authenticity? Backing (1) Claim

Backing (2) Backing (1) Backing (2) Backing 1 Backing (2) Grounds (n) Backing (2) Backing (1) 32 Qualifiers Backing 1 Chain of evidence Grounds (2) Grounds (1) Backing (1) Warrant Validity? Authenticity? Backing (1) Claim

Poor sources Backing (2) Backing (1) Backing (2) Backing 1 Backing (2) Grounds (n) Backing (2) Backing (1) 33 Qualifiers Backing 1 Chain of evidence Grounds (2) Grounds (1) Backing (1) Warrant Validity? Authenticity? Backing (1) Claim

Poor sources Backing (2) Backing (1) Backing (2) Backing 1 Backing (2) Grounds (n) Backing (2) Backing (1) Qualifiers Backing 1 Chain of evidence Grounds (2) Grounds (1) Backing (1) Warrant Validity? Authenticity? Claim Backing (1) Conclusions (1) 34

Poor sources Backing (2) Backing (1) Backing (2) Backing 1 Backing (2) Grounds (n) Backing (2) Backing (1) Backing 1 Chain of evidence Grounds (2) Grounds (1) Backing (1) Warrant Validity? Authenticity? Claim Backing (1) Conclusions (2) Conclusions (3) 35 Qualifiers

Poor sources Backing (2) Backing (1) Backing (2) Backing 1 Backing (2) Grounds (n) Backing (2) Backing (1) Backing 1 Chain of evidence Grounds (2) Grounds (1) Backing (1) Warrant Validity? Authenticity? Claim Backing (1) Conclusions (2) Conclusions (3) 36 Qualifiers

Data with “strength of evidence” below given reliability or quality threshold, indicated by red, broken borders Poor sources Backing (2) Backing (1) Backing (2) Backing 1 Backing (2) Grounds (n) Backing (2) Backing (1) Backing 1 Chain of evidence Grounds (2) Grounds (1) Backing (1) Warrant Validity? Authenticity? Backing (1) Claim “Brown worm” snaking through the argument Conclusions (1) Conclusions (2) Conclusions (3) 37 Qualifiers

Poor sources Rebuttals Backing (2) Backing (1) Backing (2) Backing 1 Backing (2) Grounds (n) Backing (2) Backing (1) Qualifiers Backing 1 Chain of evidence Grounds (2) Grounds (1) Backing (1) Warrant Validity? Authenticity? Claim Backing (1) Conclusions (2) Conclusions (3) 38 Alternative Points of View; Counter Arguments

Such Visual Representation Forms Support Thinking and Reasoning Causal Reasoning Data Frame Critical Thinking Propensity Co-variance Mutability Connect Elaborate Question Preserve Reframe Compare Purpose of thnking Question at issue Evidence Inference and interpretations Concepts Assupmtions Implications and consequences Points of view 39

Assembling Evidence? 40

Search “seattle”

seattle Search “seattle”

seattle Search “seattle”

seattle Search “seattle”

seattle Search “seattle” Search “apple”

seattle Search “seattle” Search “apple” Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua.

seattle Search “seattle” Search “apple” Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua.

seattle Search “seattle” Search “apple” Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua. Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet

seattle Search “seattle” Search “apple” Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua. Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua

seattle Search “seattle” Search “apple” Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua. Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua alliquyam

seattle Search “seattle” Search “apple” Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua. Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua alliquyam

seattle Search “seattle” Search “apple” Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua. Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua alliquyam

53

feugait nulla facilisi Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat 54 54

feugait nulla facilisi Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat 55 55

feugait nulla facilisi Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat 56 56

feugait nulla facilisi invidunt ut labore i molestie consequat ullamcorper suscipit lobortis faute Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Duis a dolor vulpu conse

molestie consequat Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat feugait nulla facilisi invidunt ut labore i ullamcorper suscipit lobortis faute Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Duis a dolor vulpu conse

feugait nulla facilisi molestie consequat Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat invidunt ut labore i ullamcorper suscipit lobortis faute Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Duis a dolor vulpu conse

feugait nulla facilisi molestie consequat Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat invidunt ut labore i ullamcorper suscipit lobortis faute Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Duis a dolor vulpu conse

feugait nulla facilisi molestie consequat Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Claim invidunt ut labore i ullamcorper suscipit lobortis faute Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Duis a dolor vulpu conse

Assembling Evidence: Design for Storytelling feugait nulla facilisi molestie consequat Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Claim … or pulled out to create a chain of evidence for (i) further analysis or (ii) to construct a claim invidunt ut labore i ullamcorper suscipit lobortis faute Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat Duis a dolor vulpu conse The bar can be hidden or shown. The annotations in the bar can be re-sequenced to create a temporal narrative or explanation …

Method of Assembly must also depend on type of inferential reasoning Moore, D. T. (2007). Critical thinking and intelligence analysis (2 nd ed. ). 63 Washington, D. C. : National Defense Intelligence College.

Collectively Reasoning workspace Hypothesis Space Conclusion Pathways -Collate, assemble, marshal -Formulation -Testing and simulation -arguments, conclusions, evidence Data Space -what’s available? -What’s changed? -Awareness: what’s in there? 64 Analysis Space -Tools and algorithms -Behaviours, relationships and patterns -what’s going on in there?

Design Challenges • Exploration under Uncertainty – Supporting search and sense-making • Revealing the nature of the data – what’s in it? – Through a keyhole • What is the structure of the data set? – How are they organised? • Relationships within and between the data – Meaningful only within the context of the goals • Constructing new assemblies of evidence – Explain, elaborate, question, reframe • Tracing analytical reasoning: Conclusion Pathways • Supporting work environment: Reasoning Workspace 65

Thank – you! Questions? Professor William Wong Dr Simon Attfield Interaction Design Center Middlesex University The Burroughs, Hendon London, U. K. w. wong @ mdx. ac. uk s. attfield idc. eis. mdx. ac. uk