Intro Urban Vis Analytics Provenance Wrapup Thinking Interactively

Intro Urban Vis Analytics Provenance Wrap-up Thinking Interactively with Visualizations Remco Chang UNC Charlotte Visualization Center

Intro Urban Vis Analytics Provenance Offline Rendering (1999 -2000) Wrap-up

Intro Urban Vis Analytics Provenance Wrap-up Offline Rendering • Master’s Thesis -- – Modeling the dynamic motion based on kinematic motion • Jiggling of muscles – “Skinnable Mesh” • Volumetric deformation – Mass-spring models • 2 nd order constraint • Approximate finite element method (FEM) with implicit integration • Took all night to render a 500 -frame (10 second) sequence • NOT at all interactive…

Intro Urban Vis Analytics Provenance Wrap-up Thinking about Interactivity in Graphics • Interactivity = – 12 frames per second appears “smooth” to most people – Or, render a frame under 0. 08 second • For complex scenes with lots of polygons – Simplify the scene – Levels of Detail (LOD)

Intro Urban Vis Analytics Provenance Wrap-up Urban Simplification (2006) • (left) Original model, 285 k polygons • (center) e=100, 129 k polygons (45% of original) • (right) e=1000, 53 k polygons (18% of original) R. Chang et al. , Legible simplification of textured urban models. IEEE Computer Graphics and Applications, 28(3): 27– 36, 2008. R. Chang et al. , Hierarchical simplification of city models to maintain urban legibility. ACM SIGGRAPH 2006 Sketches, page 130 , 2006.

Intro Urban Vis Analytics Provenance Wrap-up Urban Simplification • Which polygons to remove? Original Model Our Textured Model Simplified Model using QSlim Our Model Visually different, but quantitatively similar!

Intro Urban Vis Analytics Provenance Wrap-up Urban Simplification • The goal is to retain the “Image of the City” • Based on Kevin Lynch’s concept of “Urban Legibility” [1960] – Paths: highways, railroads – Edges: shorelines, boundaries – Districts: industrial, historic – Nodes: Time Square in NYC – Landmarks: Empire State building

Intro Urban Vis Analytics Provenance Wrap-up Urban Visualization with Semantics • How do people think about a city? – Describe New York… • Response 1: “New York is large, compact, and crowded. ” • Response 2: “The area where I live there has a strong mix of ethnicities. ” Geometric, Information, View Dependent (Cognitive)

Intro Urban Vis Analytics Provenance Wrap-up Urban Visualization • Geometric – Create a hierarchy of shapes based on the rules of legibility • Information – Matrix view and Parallel Coordinates show relationships between clusters and dimensions • View Dependence (Cognitive) – 2 D translation of yellow sphere changes the position of focus R. Chang et al. , Legible cities: Focus-dependent multi-resolution visualization of urban relationships. IEEE Transactions on Visualization and Graphics , 13(6): 1169– 1175, 2007

Intro Urban Vis Analytics Provenance Wrap-up The Role of Interaction in Visualization • We can use interactions to… [Yi et al. 2007] – – – – Select: mark something as interesting Explore: show me something else Reconfigure: show me a different arrangement Encode: show me a different representation Abstract/Elaborate: show me more or less detail Filter: show me something conditionally Connect: show me related items • In other words, we can use interactions to think.

Intro Urban Vis Analytics Provenance Wrap-up (1) Wire. Vis: Financial Fraud Analysis • In collaboration with Bank of America – Looks for suspicious wire transactions – Currently beta-deployed at Wire. Watch – Visualizes 15 million transactions over 1 year • Coordinates four perspectives: – Keywords to Accounts – Keywords to Keywords – Keywords/Accounts over Time – Account similarities (search by example)

Intro Urban Vis Analytics Provenance Wrap-up (1) Wire. Vis: Financial Fraud Analysis Heatmap View (Accounts to Keywords Relationship) Search by Example (Find Similar Accounts) Keyword Network (Keyword Relationships) Strings and Beads (Relationships over Time) R. Chang et al. , Scalable and interactive visual analysis of financial wire transactions for fraud detection. Information Visualization, 2008. R. Chang et al. , Wirevis: Visualization of categorical, time-varying data from financial transactions. IEEE VAST, 2007.

Intro Urban Vis Analytics Provenance Wrap-up (2) Bridge Analysis • Funded by the US Dept of Transportation – Commissioned after the Minnesota Bridge collapse • Current Challenges: – Bridges are inspected manually and visually (therefore subjective) – Conditions and regulations change over time – Budget changes from year to year – Need to consider external factors (traffic, population change, weather, water systems) – Which bridges should be fixed each year (or next)? • Often a repair (or reconstruction) lasts over a year

Intro Urban Vis Analytics Provenance Wrap-up (2) Bridge Analysis Dimensional Relational Geospatial Temporal R. Chang et al. , Knowledge Integrated Visual Analysis of Bridge Safety and Maintenance, SPIE 2008. R. Chang et al. , An Interactive Visual Analytics System for In-Depth Bridge Management. IEEE VAST, 2007. In submission

Intro Urban Vis Analytics Provenance Wrap-up (3) Investigative GTD • Collaboration with U. Maryland’s DHS Center of Excellence START (Study of Terrorism And Response to Terrorism) – Global Terrorism Database (GTD) – International terrorism activities from 1970 -1997 – 60, 000 incidents recorded over 120 dimensions • Visualization is designed to be “investigative” in that it is modeled after the 5 W’s: – Who, what, where, when, and [why]

Intro Urban Vis Analytics Provenance Wrap-up (3) Investigative GTD Who Where What Evidence Box Original Data R. Chang et al. , Investigative Visual Analysis of Global Terrorism, Journal of Computer Graphics Forum, 2008. When

Intro Urban Vis Analytics Provenance Wrap-up (3) Investigative GTD: Revealing Global Strategy This group’s attacks are not bounded by geo-locations but instead, religious beliefs. Y WH ? Its attack patterns changed with its developments.

Intro Urban Vis Analytics Provenance Wrap-up (3) Investigative GTD: Discovering Unexpected Temporal Pattern A geographicallybounded entity in the Philippines. Theme. River shows its rise and fall as an entity and its modus operandi. Domestic Group

Intro Urban Vis Analytics Provenance Wrap-up (4) i. PCA: Interactive PCA • Quick Refresher of PCA – Find most dominant eigenvectors as principle components – Data points are re-projected into the new coordinate system • For reducing dimensionality • For finding clusters • For many (especially novices), PCA is easy to understand mathematically, but difficult to understand “semantically”.

Intro Urban Vis Analytics Provenance Wrap-up (4) i. PCA: Interactive PCA R. Chang et al. , i. PCA: An Interactive System for PCA-based Visual Analytics. Computer Graphics Forum, 2009. To Appear.

Intro Urban Vis Analytics Provenance Wrap-up (4) Evaluation – i. PCA vs. SAS/INSIGHT • Accuracy – More Accurate – People don’t “give up” – Not faster • Overall preference – Using letter grades (A through F) with “A” representing excellent and F a failing grade.

Intro Urban Vis Analytics Provenance Wrap-up If (Interactions == Thinking)… • What is in a user’s interactions? • If (interactions == thinking), what can we learn from the user’s interactions? • Is it possible to extract “thinking” from “interactions”?

Intro Urban Vis Analytics Provenance Wrap-up What is in a User’s Interactions? Keyboard, Mouse, etc Input Visualization Human Output Images (monitor) • Types of Human-Visualization Interactions – Word editing (input heavy, little output) – Browsing, watching a movie (output heavy, little input) – Visual Analysis (closer to 50 -50)

Intro Urban Vis Analytics Provenance Wrap-up What is in a User’s Interactions? • Goal: determine if there really is “thinking” in a user’s interactions. Grad Students (Coders) Compare! (manually) Analysts Strategies Methods Findings Guesses of Analysts’ thinking Logged (semantic) Interactions Wire. Vis Interaction-Log Vis

Intro Urban Vis Analytics Provenance Wrap-up What’s in a User’s Interactions • From this experiment, we find that interactions contains at least: – 60% of the (high level) strategies – 60% of the (mid level) methods – 79% of the (low level) findings R. Chang et al. , Recovering Reasoning Process From User Interactions. IEEE Computer Graphics and Applications, 2009. To Appear. R. Chang et al. , Evaluating the Relationship Between User Interaction and Financial Visual Analysis. IEEE Symposium on VAST, 2009.

Intro Urban Vis Analytics Provenance Wrap-up What’s in a User’s Interactions • Why are these so much lower than others? – (recovering “methods” at about 15%) • Only capturing a user’s interaction in this case is insufficient.

Intro Urban Vis Analytics Provenance Wrap-up Lessons Learned • We have proven that a great deal of an analyst’s “thinking” in using a visualization is capturable and extractable. • Using semantic interaction capturing, we might be able to collect all the thinking of expert analysts and create a knowledge database that is useful for – Training: many domain specific analytics tasks are difficult to teach – Guidance: use existing knowledge to guide future analyses – Verification, and validation: go back and check to see if everything was done right. • But not all visualizations are interactive, and not all thinking is reflected in the interactions. – A model of how and what to capture in a visualization for extracting an analyst’s thinking process is necessary. – Work is currently in submission to IEEE Transactions of Visualization and Computer Graphics (part of Info. Vis 2009)

Intro Urban Vis Analytics Provenance Wrap-up Conclusion • Interactions are important for visualization and visual analysis – In considering interactions, one must be aware of the necessary speed and frame rate of the displays. • Techniques such as simplification, LOD, or approximation can be used. – Interactions have been proven to help the understanding of complex problems. • Relevant interactions have been integrated in multiple visualizations for different domains and demonstrated significant impact. – Capturing and storing analysts’ interactions have great potential • They can be aggregated to become a “knowledge database” that has traditionally been difficult to create manually.

Intro Urban Vis Analytics Provenance Wrap-up Discussion • What interactivity is not good for: – Presentation – YMMV = “your mileage may vary” • Reproducibility: Users behave differently each time. • Evaluation is difficult due to opportunistic discoveries. . – Often sacrifices accuracy • i. PCA – SVD takes time on large datasets, use iterative approximation algorithms such as online. SVD. • Wire. Vis – Clustering of large datasets is slow. Either precompute or use more trivial “binning” methods. • Interestingly, – It doesn’t save you time…

Intro Urban Vis Analytics Provenance Wrap-up Future Work • Urban Visualization – Visualizing a Semantic City. An ideal goal would be to have a “semantic” Google Map that shows more than street layouts, but describes neighborhood characteristics. – Further studies on what a “cognitive map” is and how a person gains and maintains spatial awareness. • Applying Interaction in visualizations – Pending NSF proposal applies visualization to studying science policies – NIH proposal in preparation applies visualization in monitoring and preventing sepsis. – Examine “the benefits of interactions”… • Interaction Capturing (Provenance) – Work with Pacific Northwest National Lab (PNNL) on the Active Reports project. Attempt to create an API or XML structure for capturing and storing relevant semantic interactions that is generalizable to all visualizations.

Intro Urban Vis Analytics Provenance Wrap-up In a Tribute to Randy Pausch • Here are my “head fakes” – Visualizations are very useful for analysis of complex problems with large datasets. – The Charlotte Visualization Center is a leader in the visual analytics community.

Intro Urban Vis Analytics Provenance Wrap-up Thank you! rchang@uncc. edu http: //www. viscenter. uncc. edu/~rchang

Intro Urban Vis Backup Slides Analytics Provenance Wrap-up

Intro Urban Vis Analytics Provenance Wrap-up What is in a User’s Interactions? • Approach: – Using Wire. Vis, we captured 10 financial analysts’ interactions in performing fraud detection. • Financial analysts are from Bank of America, (previous) Wachovia, etc. • Manufactured test dataset that has embedded known threat scenarios • The sessions are captured using screen-capturing, video camcorder, and semantic interaction logging. – The sessions were then converted (manually) to a text document: • High level: Strategies • Mid Level: Methods of implementation • Low Level: Findings – We then hired 4 graduate students to “interpret” the financial analysts’ interactions • Using proprietary visualizations to examine the interaction logs

Intro Urban Vis Analytics Provenance Wrap-up Journal Publications (11) • Visualization – Urban Visualization • • – Visualization and Visual Analytics • • – Dong Hyun Jeong, Caroline Ziemkiewicz, Brian Fisher, William Ribarsky, and Remco Chang. i. PCA: An interactive system for PCAbased visual analytics. Computer Graphics Forum, 2009. to appear. Remco Chang, Caroline Ziemkiewicz, Tera Marie Green, and William Ribarsky. Defining insight for visual analytics. IEEE Computer Graphics and Applications, 29(2): 14– 17, 2009. Remco Chang, Alvin Lee, Mohammad Ghoniem, Robert Kosara, William Ribarsky, Jing Yang, Evan Suma, Caroline Ziemkiewicz, Daniel Kern, and Agus Sudjianto. Scalable and interactive visual analysis of financial wire transactions for fraud detection. Information Visualization, 7: 63– 76(14), 2008. Xiaoyu Wang, Erin Miller, Kathleen Smarick, William Ribarsky, and Remco Chang. Investigative visual analysis of global terrorism database. Computer Graphics Forum, 27(3): 919– 926, 2008. Provenance • • Remco Chang, Thomas Butkiewicz, Caroline Ziemkiewicz, Zachary Wartell, Nancy Pollard, and William Ribarsky. Legible simplification of textured urban models. IEEE Computer Graphics and Applications, 28(3): 27– 36, 2008. Thomas Butkiewicz, Remco Chang, Zachary Wartell, and William Ribarsky. Visual analysis of urban change. Computer Graphics Forum, 27(3): 903– 910, 2008. Thomas Butkiewicz, Remco Chang, William Ribarsky, and Zachary Wartell. Understanding Dynamics of Geographic Domains, chapter Visual Analysis of Urban Terrain Dynamics, pages 151– 169. CRC Press/Taylor and Francis, 2007. Remco Chang, Ginette Wessel, Robert Kosara, Eric Sauda, and William Ribarsky. Legible cities: Focus-dependent multi-resolution visualization of urban relationships. Visualization and Computer Graphics, IEEE Transactions on, 13(6): 1169– 1175, Nov. -Dec. 2007. Wenwen Dou, Dong Hyun Jeong, Felesia Stukes, William Ribarsky, Heather Richter Lipford, and Remco Chang. Recovering reasoning process from user interactions. IEEE Computer Graphics and Applications, 2009. to appear Graphics, Virtual Reality, and Interface Designs • • Thomas Butkiewicz, Wenwen Dou, Zachary Wartell, William Ribarsky, and Remco Chang. Multifocused geospatial analysis using probes. Visualization and Computer Graphics, IEEE Transactions on, 14(6): 1165– 1172, Nov. -Dec. 2008. Dong Hyun Jeong, Chang Song, Remco Chang, and Larry Hodges. User experimentation: An evaluation of velocity control techniques in immersive virtual environments. Springer-Verlag Virtual Reality, 13(1): 41– 50, Mar. 2009.

Intro Urban Vis Analytics Provenance Wrap-up Conference/Workshop (19) • • • • • Wenwen Dou, Dong Hyun Jeong, Felesia Stukes, William Ribarsky, Heather Richter Lipford, and Remco Chang. Comparing usage patterns of domain experts and novices in visual analytical tasks. In ACM SIGCHI Sensemaking Workshop 2009, 2009. Xiaoyu Wang, Wenwen Dou, Rashna Vatcha, Wanqiu Liu, Shen-En Chen, Seok-Won Lee, Remco Chang, and William Ribarsky. Knowledge integrated visual analysis of bridge safety and maintenance. In Defense, Security and Sensing 2009, 2009. to appear. Xiaoyu Wang, Wenwen Dou, William Ribarsky, and Remco Chang. Integration of heterogeneous processes through visual analytics. In Defense, Security and Sensing 2009, 2009. to appear. Michael Butkiewicz, Thomas Butkiewicz, William Ribarsky, and Remco Chang. Integrating timeseries visualizations within parallel coordinates for exploratory analysis of incident databases. In Defense, Security and Sensing 2009, 2009. to appear. Thomas Butkiewicz, Dong Hyun Jeong, William Ribarsky, and Remco Chang. Hierarchical multitouch selectione techniques for collaborative geospatial analysis. In Defense, Security and Sensing 2009, 2009. to appear. Dong Hyun Jeong, Remco Chang, and William Ribarsky. An alternative definition and model for knowledge visualization. In IEEE Visualization Workshop on Knowledge Assisted Visualization, 2008. Xiaoyu Wang, Wenwen Dou, Seok won Lee, William Ribarsky, and Remco Chang. Integrating visual analysis with ontological knowledge structure. In IEEE Visualization Workshop on Knowledge Assisted Visualization, 2008. Dong Hyun Jeong, Wenwen Dou, Felesia Stukes, William Ribarsky, Heather Richter Lipford, and Remco Chang. Evaluating the relationship between user interaction and financial visual analysis. In Visual Analytics Science and Technology, 2008. VAST 2008. IEEE Symposium on, 2008. Ginette Wessel, Remco Chang, and Eric Sauda. Towards a new (mapping of the) city: Interactive, data rich modes of urban legibility. In Association for Computer Aided Design in Architecture, 2008. Ginette Wessel, Remco Chang, and Eric Sauda. Visualizing gis: Urban form and data structure. In Dietmar Froehlick and Michaele Pride, editors, Seeking the City: Visionaries on the Margins, 96 th Annual Conference of Association of Collegiate Schools of Architecture (ACSA), pages 378– 384. Association of Collegiate Schools of Architecture, 2008. Ginette Wessel, Eric Sauda, and Remco Chang. Urban visualization: Urban design and computer visualization. In CAADRIA 2008 Proceedings of the 13 th International Conference on Computer Aided Architectural Design Research in Asia, pages 409– 416, Chiang Mai, Thailand, April 9 -12, 2008. Thomas Butkiewicz, Remco Chang, Zachary Wartell, and William Ribarsky. Visual analysis for live lidar battlefield change detection. volume 6983, page 69830 B. SPIE, 2008. Josh Jones, Remco Chang, Thomas Butkiewicz, and William Ribarsky. Visualizing uncertainty for geographical information in the global terrorism database. volume 6983, page 69830 E. SPIE, 2008. Alex Godwin, Remco Chang, Robert Kosara, and William Ribarsky. Visual analysis of entity relationships in the global terrorism database. volume 6983, page 69830 G. SPIE, 2008. Thomas Butkiewicz, Remco Chang, Zachary Wartell, and William Ribarsky. Analyzing sampled terrain volumetrically with regard to error and geologic variation. volume 6495, page 64950 O. SPIE, 2007. Remco Chang, Mohammad Ghoniem, Robert Kosara, William Ribarsky, Jing Yang, Evan Suma, Caroline Ziemkiewicz, Daniel Kern, and Agus Sudjianto. Wirevis: Visualization of categorical, time-varying data from financial transactions. In Visual Analytics Science and Technology, 2007. VAST 2007. IEEE Symposium on, pages 155– 162, 30 2007 -Nov. 1 2007. Remco Chang, Thomas Butkiewicz, Caroline Ziemkiewicz, Zachary Wartell, Nancy Pollard, and. William Ribarsky. Hierarchical simplification of city models to maintain urban legibility. In SIGGRAPH ’ 06: ACM SIGGRAPH 2006 Sketches, page 130, New York, NY, USA, 2006. ACM. Remco Chang, Robert Kosara, Alex Godwin, and William Ribarsky. Towards a role of visualization in social modeling. AAAI 2009 Spring Symposium on Technosocial Predictive Analytics, 2009. to appear. Ginette Wessel, Eric Sauda, and Remco Chang. Mapping understanding: T ransforming topographic maps into cognitive maps. Geo. Vis Hamburg Workshop, 2009.

Intro Urban Vis Analytics Provenance Wrap-up Grants Awarded (2) • NSF Sci. SIP: – A Visual Analytics Approach to Science and Innovation Policy. • PI: William Ribarsky, Co-PIs: Jim Thomas, Remco Chang, Jing Yang. • $746, 567. 2009 -2012 (3 years). • DHS International Program: – Deriving and Applying Cognitive Principles for Human/Computer Approaches to Complex Analytical Problems. • PI: William Ribarsky, Co-PIs: Brian Fisher, Remco Chang, John Dill. • $200, 000. 2009 -2010 (1 year). • In Submission / Preparation – 2 other NSF proposals are pending reviews – 1 NSF and 1 NIH proposals are currently under preparation

Intro Urban Vis Analytics Provenance Wrap-up Professional Activities • Committee / Panelists – Committee Member: AAAI Spring-09 Symposium on Technosocial Predictive Analytics – Panelist: 3 rd Annual DHS University Summit. Panel: Research to Reality – 3 rd Annual DHS University Summit. Panel: Visual Analytics and Discrete Science Integration into the DHS Center of Excellence Program • Invited Talks – – – Dec 13, 2006 Google Inc. Simplification of Urban Models based on Urban Legibility July 6, 2007 Naval Research Lab. Urban Visualization Oct 4, 2007 Charlotte Viscenter. Urban Visualization Oct 17, 2007 Charlotte Metropolitan GIS Users Group. GIS and Urban Visualization Nov 19, 2007 START Center at University of Maryland. Integrated Visual Analysis of the Global Terrorism Database Nov 29, 2007 Charlotte Viscenter. Integrated Visual Analysis of the Global Terrorism Database Jan 25, 2008 Do. D/DHS Social Science Modeling and Information Visualization Symposium. Social Science and Information Visualization on Terrorism and Multimedia May 14, 2008 Charlotte Metropolitan GIS User Group. Multi-Focused Geospatial Analysis Using Probes Aug 27, 2008 Do. D/DHS Symposium for Overcoming the Information Challenge in Federated Analysis: From Concept to Practice. Roadmap of Visualization Mar 19, 2009 DHS University Summit. Panel: Research to Reality Mar 19, 2009 DHS University Summit. Panel: Visual Analytics and Discrete Science Integration into the DHS Center of Excellence Program

Intro Urban Vis Analytics Provenance Wrap-up Introduction • The common thread between all the journals and conference papers, talks, grants, etc. is the use of interactivity in graphics, visualization, and visual analytics. • This presentation will focus on select projects and publications due to time constraint. For more information, see my website: www. viscenter. uncc. edu/~rchang