Visual Analytics for Big Video Visualization Robert S

Visual Analytics for Big Video Visualization Robert S. Laramee Visual and Interactive Computing Group Department of Computer Science Swansea University R. S. Laramee@swansea. ac. uk Robert S. Laramee r. s. laramee@swansea. ac. uk 1 1

Overview Data Visualization Motivation: Rugby Sport Video Analysis Visualization Using Glyphs Glyph Design and Sorting System Interface Knowledge Assisted Ranking Conclusions and Acknowledgments Robert S. Laramee r. s. laramee@swansea. ac. uk 2 2

Visualization and the Visual Cortex Visualization exploits our powerful visual system 2 million nerve fibers coming from optic nerves Several billion neurons devoted to analyzing visual information (30% cortex) 8% for touch, 3% for hearing (Discover, 1993, Ware, 2013) Enables massively parallel processing of the visual field, i. e. , incoming color, motion, texture, shapes etc. Robert S. Laramee 3 r. s. laramee@swansea. ac. uk

Benefits of Data Visualization Robert S. Laramee r. s. laramee@swansea. ac. uk 4 4

Visual Analytics ”Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces. ” –Illuminating the Path: The Research and Development Agenda for Visual Analytics, Thomas and Cook, 2005 Robert S. Laramee r. s. laramee@swansea. ac. uk 5 5

Motivation: Massive Multimedia Data from Videos Case Study-Sport Videos: Rugby Sport Video Analysis Trying to find most important events Trying to look for patterns and relationships Trying to compare games Watching video is time consuming Robert S. Laramee r. s. laramee@swansea. ac. uk 6 6

Visualization Using Glyphs Data Visualization Using Glyphs: a special class of visualization Discrete: The underlying data is represented pictorially Glyphs by themselves are not novel Sorting: we combine visualization and computer vision techniques, add sorting and interaction capabilities Robert S. Laramee r. s. laramee@swansea. ac. uk 7

Challenges of Sport Video Data Robert S. Laramee r. s. laramee@swansea. ac. uk 8

Glyphs For Visualizing Rugby Events Robert S. Laramee r. s. laramee@swansea. ac. uk 9

Glyphs For Visualizing Rugby Events Robert S. Laramee r. s. laramee@swansea. ac. uk 10

Visualization Pipeline Robert S. Laramee r. s. laramee@swansea. ac. uk 11

System Interface Robert S. Laramee r. s. laramee@swansea. ac. uk 12

Hierarchical Sorting Robert S. Laramee r. s. laramee@swansea. ac. uk 13

Glyph Sorting Results and Demo 2 matches Events are: Gain (Y) vs Event (X) Plotted Kick Reception, Restart Reception, Lineout, Turnover, Scrum, Penalty • Purple highlights indicate points scored Robert S. Laramee r. s. laramee@swansea. ac. uk 14

Knowledge-Assisted Ranking Robert S. Laramee r. s. laramee@swansea. ac. uk 15 Rugby Sport Video Analysis Trying to find most important events (manual search), looking for patterns, relationships, game comparisons Analysis: Trying to rank events according to their importance Challenging when incorporating several data dimensions Watching video is time consuming (linear search) 15

Enhanced System Overview Glyph-Based Visualization: shows events sorted by ranking and Y axis (Gain) Model Visualization: Analysis how current model parameters and accuracy correspond to ranking input Ranking Input: ranking model can be exported to primary axis in glyph-based visualization Glyph Control Panel: interaction enables control of axes within glyph-based visualization Robert S. Laramee r. s. laramee@swansea. ac. uk 16

Visual Analytic Pipeline Tacit Knowledge: A priori knowledge that domain experts have Partial Knowledge: Knowledge domain experts have about most important influential data attributes (or sort keys) Tacit Knowledge does not scale up to large number of events, but choosing a few representative events is easy System Knowledge: No a priori knowledge - derived from tacit knowledge + partial knowledge + sorting functions Robert S. Laramee r. s. laramee@swansea. ac. uk 17

Event Ranking Using Regression Approach inspired by card sorting (Rugg and Mc. George 1997) • • User-centered approach to categorize a set of items in to groups, e. g. , symbols in cartography (Roth et al. , 2011), online course sites (Doubleday, 2013) “Cards” are glyphs Let: e 1, e 2, …, en be events Let: es 1<es 2<…<esn an ordering Define: y = Eb where E is an n x m matrix, and bj are the weights or contributions of each sort key Goal: estimate weights of b and ranking function using regression Robert S. Laramee r. s. laramee@swansea. ac. uk 18

Event Ranking Models User provides sample event ranking (small number e. g. , 9) Robert S. Laramee r. s. laramee@swansea. ac. uk • System compares manual ranking with ranking predicted by regression model • • Each polyline is an event • • Last axis encodes ranking confidence • Performance of each model is evaluated, best model is chosen 19 Contribution of each data attribute within model is depicted by gauges on each axis Each regression model may discover a different set of key performance indicators

Refining Model Parameters User may rank event based on an ad hoc requirement, intuition, guesses Robert S. Laramee r. s. laramee@swansea. ac. uk 20 • Thus user may refine model parameters by applying weighting parameters to sort keys • Enables user to explore new sorting strategies and understand impact on predicted ranking • e. g. remove a sort key

Interactive Brushing All views are coordinated and linked. Robert S. Laramee r. s. laramee@swansea. ac. uk 21 • Glyphs corresponding to brushed polylines are highlighted (in focus) • Context glyphs can be scaled down • Users can select a glyph and replay original video of event

Comparing Two Matches • • Comparison of 2 matches: 81 -7, 16 -17 Expert user starts by selecting representative glyphs based on Gain Events are input into ranking input and confirm by watching corresponding video Analysts then visually assess resulting model Phases are removed from attribute ranking Large cluster of scoring events is observed in 1 st match and absent in 2 nd Two poor kicks + turnovers discovered quickly Robert S. Laramee r. s. laramee@swansea. ac. uk 22

Knowledge Assisted Ranking Results and Demo Analyst: “Using the software has enabled us to discover new key performance indicators that we wouldn’t have recognized before…” Coach: “The system here is a good way at grouping clips…” Player: “The software is useful as it allows you to break up the gaim by what you want to see…” Robert S. Laramee r. s. laramee@swansea. ac. uk 23

Acknowledgements Thank you for your attention! Any questions? We thank the following: David H. . S. Chung, Rhodri Bown, Min Chen, Iwan W. Griffiths, Phillip A. Legg, Adrian Morris, Matthew L. Parry, The Welsh Assembly Government (WAG) For more information, please see: David Chung, Phillip Legg, Matthew Parry, Iwan Griffiths, Rhodri Bown, Robert S Laramee, and Min Chen, Knowledge-Assisted Ranking: A Visual Analytic Application for Sport Event Data, IEEE Computer Graphics & Applications (IEEE CG&A), forthcoming Robert S. Laramee r. s. laramee@swansea. ac. uk 24

Empirical User Study Investigate difficulty of formalizing a ranking. 5 participants Task: identify and rank set of events by most important positive outcome Task 1: 5 events Assess confidence Task 4: specify model Task 2: 10 events Color map emphasizes worst and best events Task 3: identify most important set of influential attributes Robert S. Laramee r. s. laramee@swansea. ac. uk 25