Opening the Black Box of Interaction in Visualization

Opening the Black Box of Interaction in Visualization Hans-Jörg Schulz 1, Tatiana v. Landesberger 2, Dominikus Baur 3 VIS Tutorial 2014 D 1. 2. 3. Fraunhofer IGD, Rostock, Germany TU Darmstadt, Germany Dominikus Baur Interfacery . MINIK. US

PART I: INTERACTION ACTIVITIES Speaker: Tatiana von Landesberger VIS Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 2

Part 1: Interaction activities Activities: What the user does to trigger a change in the computer (Action) Metaphor: What the user thinks the computer is doing and vice versa (Understanding) Architecture: What the computer actually does (Reaction) Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 3

Part 1: Interaction activities Activities: What the user does to trigger a change in the computer (Action) Metaphor: What the user thinks the computer is doing and vice versa (Understanding) Architecture: What the computer actually does (Reaction) Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 4

Overview of Part 1 • 6 W’s of User’s Interaction and interaction loop • Systematization of interaction • Human (Ws) and System (Vis) • Vis/VA-focused systematizations: 1. Visualization, 2. Visual Data Mining 3. Reasoning • Third view: Interaction Support Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 5

Motivation • System developers: • What to include in my system • Researchers: • What is there and what is missing • Users: • What to expect from the system • Developers, researchers, …: • Canonicum for evaluation and system testing Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 6

Make it easier for you… Motivation Systematization of perspectives … Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 7

USER’S ACTION AND THE INTERACTION LOOP Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur http: //www. smartgirl. org/brain-food/career-hubs/human-computer-interaction. png What is interaction from user’s point of view? 8

6 Ws of Interaction WHY do we interact? What is the goal? WHAT is the purpose? What is the intended effect of interaction? Effects and means HOW do we interact? Which means do we use/have at disposal? WHO interacts? Who are the users interacting? What is their background? WHEN do we interact? Context of interaction When is interaction needed? WHERE is interaction used? Where users interact? Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [adapted & merged Roth 13, Jansen et al 13] 9

6 Ws of Interaction WHY do we interact? What is the goal? WHAT is the purpose? What is the intended effect of interaction? Effects and means HOW do we interact? Which means do we use/have at disposal? WHO interacts? Who are the users interacting? What is their background? WHEN do we interact? Context of interaction When is interaction needed? WHERE is interaction used? Where users interact? Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [adapted & merged Roth 13, Jansen et al 13] 10

6 Ws of Interaction WHY do we interact? What is the goal? WHAT is the purpose? What is the intended effect of interaction? Effects and means HOW do we interact? Which means do we use/have at disposal? WHO interacts? Who are the users interacting? What is their background? WHEN do we interact? Context of interaction When is interaction needed? WHERE is interaction used? Where users interact? Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [adapted & merged Roth 13, Jansen et al 13] 11

6 Ws of Interaction WHY do we interact? What is the goal? WHAT is the purpose? Effects and means What is the intended effect of interaction? HOW do we interact? Which means do we use/have at disposal? WHO interacts? Who are the users interacting? What is their background? WHEN do we interact? Context of interaction When is interaction needed? WHERE is interaction used? Where users interact? [adapted & merged from Roth 13, Jansen et al 13] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 12

Hierarchic View on Interaction WHY do we interact? WHAT is the purpose? HOW do we interact? Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 13

Norman’s Model of Interaction 8. Take further action (compare outcome with goal) EXECUTION 1. Establish a goal (Why? ) Execution/ 2. Form intention/identify task Evaluation loop (What? ) 3. Specify action sequence (How? ) 4. Execute action EVALUATION 7. Evaluate the outcome (Why? ) 6. Interpret the system’s state (What? ) 5. Perceive the state of the system (How? ) [Norman 88] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 14

Preliminary Summary 1. 6 Ws of interaction: 1. Effects+means: Why? , What? How? 2. Context: Who? Where? When? 2. Hierarchic nature of interaction 3. Execution/Evaluation loop Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 15

Which interactions exist? INTERACTION SYSTEMATIZATION Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 16

TWO VIEWS ON INTERACTION Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 17

What is interaction Systematization 1. 6 Ws of interaction: 1. Effects+means: Why? , What? How? 2. Context: Who? Where? When? 2. Hierarchic nature of interaction 3. Execution/Evaluation loop Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 18

What is interaction Systematization 1. 6 Ws of interaction: 1. Effects+means: Why? , What? How? 2. Context: Who? Where? When? 2. Hierarchic nature of interaction 3. Execution/Perception loop Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur User-centric view 19

What is interaction Systematization 1. 6 Ws of interaction: 1. Effects+means: Why? , What? How? 2. Context: Who? Where? When? 2. Hierarchic nature of interaction 3. Execution/Perception loop User-centric view Visualization side? Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 20

Human ↔ Visualization WHY WHAT HOW Subjective perception Visualization changes What should be modified in the view (goal/intention) What in the visualization is modified How it should be modified (which action) How this is done (software/hardware) [adjusted Roth 13] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 21

Human ↔ Visualization WHY WHAT [Card et al. 99] HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 22

Human ↔ Visual Analytics WHY View manipulation Visualization Mapping Transformation Data WHAT Model buildin g Model visualizatio n Models Knowledge Parameter refinement Information mining HOW Feedback loop Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Keim et al. 2008] 23

Information Visualization Model [Card et al. 99] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 24

Information Visualization Model Data Visual [Card et al. 99] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 25

Visualization Visual Analytics Simple Information Visualization Model Visualization View Manipulation Mapping Transformation Data Knowledge [Keim et al. 2008] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 26

Visualization Visual Analytics Simple Data Mining Model Transformation Data Knowledge Data Mining/ Models Parameter refinement Information mining Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Keim et al. 2008] 27

Visual Analytics Model View manipulation Visualization Mapping Transformation Data Model building Model visualization Models Knowledge Parameter refinement Information mining Feedback loop Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Keim et al. 2008] 28

3 Ways of Visual Analytics Way 1: Info. Vis View manipulation Visualization Mapping Transformation Data Model building Model visualization Models Knowledge Parameter refinement Information mining Feedback loop Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 29

3 Ways of Visual Analytics Way 2: Visual Data Mining View manipulation Visualization Mapping Transformation Data Model building Model visualization Models Knowledge Parameter refinement Information mining Feedback loop [Keim et al. 2008] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 30

3 Ways of Visual Analytics Way 3: Provenance/Sensemaking/Reasoning View manipulation Visualization Mapping Transformation Data Model building Model visualization Models Knowledge Parameter refinement Information mining Feedback loop [Keim et al. 2008] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 31

Interaction Need in Visual Analytics Support all 3 ways via visual means View manipulation Visualization Mapping Transformation Data Model building Model visualization Models Knowledge Parameter refinement Information mining Feedback loop [Keim et al. 2008] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 32

Systematization of Interaction 2. Visual Data Mining WHY [Bertini & Lalanne 09] [Shneiderman 96] […] 1. Information Visualization [Keim 02] WHAT [Pike et al 09] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Roth 13] [Wybrow et al 14] [Zhou & Fesner 98] [Jansen, Dragicevic 13] […] 3. Reasoning [Heer & Shneiderman 12] [Gotz & Zhou 08] HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Kerren & Schreiber 12] […] 33

Systematization of Interaction 2. Visual Data Mining WHY [Bertini & Lalanne 09] [Shneiderman 96] […] 1. Information Visualization [Keim 02] WHAT [Pike et al 09] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Roth 13] [Wybrow et al 14] [Zhou & Fesner 98] [Jansen, Dragicevic 13] […] 3. Reasoning [Heer & Shneiderman 12] [Gotz & Zhou 08] HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Kerren & Schreiber 12] […] 34

Levels of Systematization: Example WHY [Yi et al 07] WHAT ? Select, Explore, Reconfigure, Encode, Abstract/Elaborate, Filter, Connect HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 35

Levels of Systematization: Problem of ambiguous terms WHY [Yi et al 07] WHAT ? Select, Explore, Reconfigure, Encode, Abstract/Elaborate, Filter, Connect What/where in the pipeline? HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 36

Pipeline-Focused Interaction Systematization 2. Visual Data Mining [Shneiderman 96] 1. Information Visualization [Keim 02] [Pike et al 09] [Roth 13] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Bertini & Lalanne 09] […] 3. Reasoning [Wybrow et al 14] [Zhou & Fesner 98] [Heer & Shneiderman 12] [Jansen, Dragicevic 13] [Gotz & Zhou 08] […] [Kerren & Schreiber 12] […] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 37

Pipeline-Focused Interaction Systematization 2. Visual Data Mining [Shneiderman 96] 1. Information Visualization [Keim 02] [Pike et al 09] [Roth 13] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Bertini & Lalanne 09] ? […] 3. Reasoning [Wybrow et al 14] [Zhou & Fesner 98] [Heer & Shneiderman 12] [Jansen, Dragicevic 13] [Gotz & Zhou 08] […] [Kerren & Schreiber 12] […] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 38

Unified VA Interaction Systematization 1. Visualization 2. Data mining 3. Reasoning Data changes • Selection • Editing • Analytic process tracking • Editing (annotation) Visualization changes Data mining changes • Scheme: type and mapping • Parameters • Scheme: data processing type • Parameters Reasoning changes • Scheme: change analysis type • Parameters [von Landesberger et al. 2014] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 39

View manipulation Visualization Mapping Transformation Data Model building Model visualization Models Knowledge Parameter refinement Information mining Feedback loop 3 Ways of Visual Analytics: Info. Vis (Way 1) INFOVIS – FOCUSED SYSTEMATIZATIONS Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 40

1. Visualization Systematizations [Shneiderman 96] 1. Information Visualization [Keim 02] [Pike et al 09] [Roth 13] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Wybrow et al 14] [Jansen, Dragicevic 13] [Zhou & Fesner 98] Infovis pipeline based systematization […] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 41

1. Visualization Systematization Text #Words #Sentence 50 9 100 20 80 7 #Words Size #Senten ces Color [Card et al 99] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur

Info. Vis Interaction: View Transformation Text #Words #Sentence 50 9 100 20 80 7 #Words Size #Senten ces Color [Card et al 99] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur

Info. Vis Interaction: View Transformation • Navigation • Pan, zoom, scroll, . . . • Highlighting Hover Select+highlight Navigation in visible space Brushing and linking Magic lenses View reconfiguration (Re-)arrange multiple views on the screen Open/close new views Source: maps. google. com Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 44

Info. Vis Interaction: View Transformation NAVIGATE: Paris Sydney • Navigation • Pan, zoom, scroll, . . . • Highlighting Hover Select+highlight Navigation in visible space Brushing and linking Magic lenses View reconfiguration (Re-)arrange multiple views on the screen Open/close new views Source: maps. google. com Problem: Cumbersome, time consuming, “lost in space” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 45

Info. Vis Interaction: View Transformation Navigation Video 01: Topology-Aware Navigation in Large Networks [Moskovich et al. 09] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 46

Info. Vis Interaction: View Transformation • Navigation Pan, zoom, scroll, . . . • • Highlighting • • Hover Select+highlight Brushing and linking Magic lenses • View reconfiguration • • (Re-)arrange multiple views on the screen • Highlighting important information Open/close new s Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 47

Info. Vis Interaction: View Transformation • Navigation Pan, zoom, scroll, . . . • • Highlighting • • Hover Select+highlight Brushing and linking Magic lenses • View reconfiguration • • (Re-)arrange multiple views on the screen Open/close new views • Configuring multiple views Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 48

Info. Vis Interaction: Visual Mapping Text #Words #Sentence 50 9 100 20 80 7 #Words Size #Senten ces Color [Card et al 99] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 49

Info. Vis Interaction: Visual Mapping • Visualization type • Type of visualization • Scatterplot/matrix • Node-link/matrix • Type of mapping • E. g. color/size/form Video 10 [van den Elzen & van Wijk 13] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 50

Info. Vis Interaction: Visual Mapping • Visualization type • Type of visualization • Type of mapping • Mapping parameter • Data mapping • E. g. color scheme Video 10 [van den Elzen & van Wijk 13] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 51

Info. Vis Interaction: Visual Mapping • Visualization type • Type of visualization • Type of mapping • Mapping parameter • Data mapping • E. g. color scheme Video 10 [van den Elzen & van Wijk 13] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 52

Info. Vis Interaction: Visual Mapping • Visualization type • Type of visualization • Type of mapping • Mapping parameter • Data to be mapped • E. g. color scheme • Further specifics • E. g. type of layout, sorting Video 10 [van den Elzen & van Wijk 13] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 53

Info. Vis Interaction: Data Manipulation Text #Words #Sentence 50 9 100 20 80 7 #Words Size #Senten ces Color [Card et al 99] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 54

Info. Vis Interaction: Data Manipulation • Data navigation • drill down, expand, filter, … • Data transformation • Change data values by calculation • Normalization, aggregation, … • Data editing • Change data values by editing • Create data Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 55

Info. Vis Interaction: Data Manipulation • Data navigation • drill down, expand, filter, … • • Data transformation Top down • • Change data on values by calculation Filter, details demand • • Normalization, aggregation, … Bottom up • • Data editing Expand on demand • • Change data values by editing Middle out • • Create Start indata the middle [von Landesberger et al 11] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 56

Info. Vis Interaction: Data Manipulation • Data navigation • Search, Show Context, Expand on Demand drill down, expand, filter, … • • Data transformation Top down • • Change data on values by calculation Filter, details demand • • Normalization, aggregation, … Bottom up • • Data editing Expand on demand • • Change data values by editing Middle out • • Create Start indata the middle [van Ham & Perer 09] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 57

Info. Vis Interaction: Data Manipulation Video 02: data search and expand Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 58

Info. Vis Interaction: Data Manipulation • Data navigation • drill down, expand, filter, … • • • Normalization (lin, log, exp, . . ) Aggregation (manual, according to data, …) … • Data editing • • Change data values by editing Create data Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur Source: Gapminder. org • Data transformation 59

Info. Vis Interaction: Data Manipulation • Data navigation drill down, expand, filter, … • • Data transformation • Normalization (e. g. lin, log) • Aggregation • • Manual • According to data attributes • According to data structure (e. g. communities) Etc. Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 60

Info. Vis Interaction: Data Manipulation Data navigation • • drill down, expand, filter, … Data transformation • • Normalization (lin, log, exp, . . ) • Aggregation (manual, according to data, …) Data editing • • Change values • Create data • Individual values • Whole datasets Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 61

Info. Vis Interaction: Data editing Video 03: edit Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 62

PC/DC On the Highway to Data [Bremm et al 2012] Info. Vis Interaction: Data creation Video 04: create [Bremm et al 2012] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 63

Summary: Info. Vis Interaction Text #Words #Sentence 50 9 100 20 80 7 #Words Size #Senten ces Color

Visualization View manipulation Mapping Transformation Data Model building Model visualization Models Information mining Knowledge Parameter refinement Way 2: Visual data mining VISUAL DATA MINING INTERACTION Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 65

VA Interaction Systematization 2. Visual Data Mining [Bertini & Lalanne 09] 1. Information Visualization […] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 66

Visual Data Mining • Computationally enhanced Visualization (V++) • Visually enhanced Mining (M++) • Integrated Visualization and Mining (VM) [Bertini & Lalanne 09] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 67

Visual Data Mining Interaction • Manipulating and tuning: Vis: changing representation parameters DM: changing model parameters • Changing the scheme: Vis: changing the visual mapping or visual representation DM: changing the data model [Bertini & Lalanne 09] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 68

Visual Data Mining Interaction • Manipulating and tuning: Vis: changing representation parameters For example: changing color scheme DM: changing model parameters For example which motif is searched Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 69

Visual Data Mining Interaction • Changing the scheme : Vis: changing the visual mapping or visual representation For example: changing node-link diagram to adjacency matrix DM: changing the data model For example motif search vs clustering Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 70

Visual Data Mining Interaction: Motif search and Visualization Video 05: data mining [von Landesberger et al 09] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 71

Visualization View manipulation Mapping Transformation Data Model building Knowledge Model visualization Models Information mining Parameter refinement [Keim et al. 2008] Way 3: Feedback loop SENSEMAKING, PROVENANCE, REASONING Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 72

VA Interaction Systematization 2. Visual Data Mining 1. Information Visualization 3. Reasoning [Heer & Shneiderman 12] [Gotz & Zhou 08] [Kerren & Schreiber 12] […] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 73

Reasoning/Provenance Systematization Visualization Provenance [Heer & Shneiderman 12] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 74

Reasoning/Provenance Systematization • Insight • Visualization Annotate, Bookmark • History • Record • Guide • Provenance • Cooperate • [Heer & Shneiderman 12] redo, undo, revisit Share Combined [Gotz & Zhou, Kerren et al, 75 Heer & Shneiderman 12]

Reasoning/Provenance Visualization Visual history & annotation [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 76

Reasoning/Provenance Visualization Visual history & annotation [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 77

Reasoning/Provenance Visualization Visual history & annotation [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 78

Reasoning/Provenance Visualization Visual history & annotation [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 79

Reasoning/Provenance Visualization Visual history & annotation Same interaction type aggregation [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 80

Reasoning/Provenance Visualization Visual history & annotation Same interaction type aggregation using interaction systematization [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 81

Reasoning/Provenance Visualization Visual history & annotation Aggregated interaction [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 82

Reasoning/Provenance Visualization Visual history & annotation [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 83

Reasoning/Provenance Visualization Visual history & annotation “Go back” - review/revise [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 84

Reasoning/Provenance Visualization Visual history & annotation “Go back” - review/revise [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 85

Reasoning/Provenance Visualization Visual history & annotation Start a different exploration path [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 86

Reasoning/Provenance Visualization Visual history & annotation Start a different exploration path [von Landesberger et al 14] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 87

Systematization of Interaction 2. Visual Data Mining WHY [Bertini & Lalanne 09] [Shneiderman 96] […] 1. Information Visualization [Keim 02] WHAT [Pike et al 09] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Wybrow et al 14] [Zhou & Fesner 98] [Jansen, Dragicevic 13] 3. Reasoning [Roth 13] […] [Heer & Shneiderman 12] [Gotz & Zhou 08] HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Kerren & Schreiber 12] […] 88

Systematization of Interaction - according to Info. Vis Pipeline 2. Visual Data Mining WHY [Bertini & Lalanne 09] [Shneiderman 96] […] 1. Information Visualization [Keim 02] WHAT [Pike et al 09] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Wybrow et al 14] [Zhou & Fesner 98] [Jansen, Dragicevic 13] 3. Reasoning [Roth 13] […] [Heer & Shneiderman 12] [Gotz & Zhou 08] HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Kerren & Schreiber 12] […] 89

Info. Vis-Focused Systematization: Problem of ambiguous terms WHY WHAT HOW ? Why? What for? How exactly? “Change mapping” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 90

SYSTEMATIZATION: INTERACTION SUPPORT Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 91

Interaction hierarchy WHY WHAT HOW [Sedic & Parsons 10] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 92

Interaction hierarchy Time & cognitive burden WHY WHAT HOW [Sedic & Parsons 10] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 93

Three Dimensions: Human rt o p up s n o ti c a er Int Visualization Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 94

Interaction support • Slow/cumbersome • High cognitive burden • Full control “manual” “DOI-based” “supported” “data-driven” • Fast • No cognitive burden • No control “smart” “automatic” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 95

Interaction support • Slow/cumbersome • High cognitive burden • Full control “manual” “DOI-based” “supported” “data-driven” • Fast • No cognitive burden • No control “smart” “automatic” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 96

Interaction support Video 06: edgelens Supported • Snap to grid • Edgelens “DOI-based” “supported” “data-driven” “smart” “manual” “automatic” [Wong & Carpendale 03] Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 97

Interaction support Data driven • Topology-aware navigation • Data-aware selection • Slow/cumbersome • High cognitive burden • Full control “manual” Video 07: data driven “DOI-based” “supported” • Fast • No cognitive burden • No control “smart” “data-driven” “automatic” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 98

Interaction support Video 08: guidance • Slow/cumbersome • High cognitive burden • Full control “manual” Guidance • Small multiples • DOI-based exploration “DOI-based” Guidance “supported” • Fast • No cognitive burden • No control “smart” “data-driven” “automatic” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 99

Interaction support Smart • Tableau “show me” • Slow/cumbersome • High cognitive burden • Full control “manual” “DOI-based” “supported” • Fast • No cognitive burden • No control “smart” [Heer et al 08] “data-driven” “automatic” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 100

Interaction support Video 09: automatic • Slow/cumbersome • High cognitive burden • Full control “manual” Automatic • Node. Trix “DOI-based” “supported” • Fast • No cognitive burden • No control “smart” “data-driven” “automatic” Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 101

Part 1: Interactions SUMMARY Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 102

Summary 2. Visual Data Mining WHY [Bertini & Lalanne 09] [Shneiderman 96] […] 1. Information Visualization [Keim 02] WHAT [Pike et al 09] [Parsons & Sedig 14] [Dix & Ellis 98] [Spence 07] [Yi et al. 07] [Wybrow et al 14] [Zhou & Fesner 98] [Jansen, Dragicevic 13] 3. Reasoning [Roth 13] […] [Heer & Shneiderman 12] [Gotz & Zhou 08] HOW Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur [Kerren & Schreiber 12] […] 103

THANK YOU Q&A Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 104

Next: Interaction architecture Hans-Jörg Schulz Activities: What the user does to trigger a change in the computer (Action) Metaphor: What the user thinks the computer is doing and vice versa (Understanding) Architecture: What the computer actually does (Reaction) Vis Tutorial: Opening the Black Box of Interaction in Visualization – H. -J. Schulz, T. v. Landesberger, D. Baur 105