Ken Hinckley Shengdong Zhao Raman Sarin Patrick Baudisch

  • Slides: 44
Download presentation
Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman & Desney

Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman & Desney Tan oronto

In situ search for digital ink Ink. Seine Ink + Seine : -n: a

In situ search for digital ink Ink. Seine Ink + Seine : -n: a fishing net -v: to fish with a seine = insanely cool ink search! Ink. Seine lets the user fish for useful information directly from ink notes. Fishermen catching salmon on the Columbia River using a seine. (courtesy Wikipedia)

Active note taking system for active reading (reading + critical thinking + learning) XLibris—pen-based

Active note taking system for active reading (reading + critical thinking + learning) XLibris—pen-based Ink. Seine combines pen-based note taking with searching, linking, collecting & sensemaking:

Core interaction pattern: Ink -Search-Gather r ch ar the Se Ink Ga – excels

Core interaction pattern: Ink -Search-Gather r ch ar the Se Ink Ga – excels at capturing user’s thoughts & ideas with minimum distraction from formatting issues Ink

Ink-Search-Gather interaction pattern Search – digging through the file system on a tablet is

Ink-Search-Gather interaction pattern Search – digging through the file system on a tablet is really a pain ch ar Se r the Ga Active note taking benefits from low overhead access to supporting materials & related docs Focus on personal search, but support Web as well Ink

Ink-Search-Gather interaction pattern Gather ch ar Se r the Thumbnail hyperlinks to documents Clippings

Ink-Search-Gather interaction pattern Gather ch ar Se r the Thumbnail hyperlinks to documents Clippings from documents Queries that can be revisited later Ink Ga – Make it easy to use what you find by pulling content directly into unstructured notes

Ink-Search-Gather interaction pattern Ink. Seine ar Se r the ch Ink Ga unifies these

Ink-Search-Gather interaction pattern Ink. Seine ar Se r the ch Ink Ga unifies these three elements in a fluid user experience that is integrated directly with inking

In situ search Leverage preexisting ink to initiate search No context switch to “search

In situ search Leverage preexisting ink to initiate search No context switch to “search app” – stay in the flow No tedious copying of information to a “search box” Queries as first class objects, commingled with ink notes Flag some ink for search for later Visible and salient search history as part of notes Queries can be copied, pasted, moved, etc. Interleave inking, searching & gathering No barrier between inking / searching – note ideas while searching, side-by-side searching, span app boundaries for content gathering Tightly couple queries with application content Queries naturally persist in originating context Easy re-visitation of prior queries Results of queries become new content

Rich tradeoffs in cost structure of sensemaking Continuum of time cost – can defer

Rich tradeoffs in cost structure of sensemaking Continuum of time cost – can defer at any point: 1 second: Capture thought “I should search on this” 5 seconds: Trigger search & see initial results list 10 s: Open a document from result list, or drag result into notes 10 s-30 s: Inspect details of a few results Apply filters to results Add a word to the query Grab a snapshot from a short document Correct occasional reco errors – but Tablet’s reco is amazingly good! Longer: sufficient depth for common info needs; not mired by last 10% of complex queries that require full featured search tools Scroll through longer results lists Inspect result documents to see if they meet information needs Sideways searches based on Title, Date, or Author of a result

Ink. Seine system VIDEO

Ink. Seine system VIDEO

Paper Prototype

Paper Prototype

Paper Prototype

Paper Prototype

Paper Prototype

Paper Prototype

Findings Existing desktop Difficult text entry search tool difficult to use Enter and correct

Findings Existing desktop Difficult text entry search tool difficult to use Enter and correct text

Findings Existing desktop Difficult text entry search tool difficult to use Enter and correct

Findings Existing desktop Difficult text entry search tool difficult to use Enter and correct text Annoying context switch and flow interruption Switch among apps and context

Findings Existing desktop Difficult text entry search tool difficult to use Enter and correct

Findings Existing desktop Difficult text entry search tool difficult to use Enter and correct text Annoying context switch and flow interruption Switch among apps and context Tedious information gathering Difficult to link to documents and gather pieces of information

Notebook Control

Notebook Control

In Situ Search

In Situ Search

Breadcrumb

Breadcrumb

Breadcrumb

Breadcrumb

Trigger Personal Search

Trigger Personal Search

Search Panel

Search Panel

Query Area

Query Area

Result List

Result List

Result List Focus Result

Result List Focus Result

File and Timeline Filters

File and Timeline Filters

Gestures for Filters Tap - Select Cross – Multi-select Disjunctive cross – Multi-select Cross

Gestures for Filters Tap - Select Cross – Multi-select Disjunctive cross – Multi-select Cross & Turn Right – Exclude

Work Flow & Occlusion Selection Correction Search

Work Flow & Occlusion Selection Correction Search

Work Flow & Occlusion

Work Flow & Occlusion

Work Flow & Occlusion

Work Flow & Occlusion

Crossing Widgets

Crossing Widgets

Resize Search Panel

Resize Search Panel

Examine Document

Examine Document

Document & Tracking Menu

Document & Tracking Menu

Multiple Search Panels

Multiple Search Panels

First Round Observations Gesture discovery problem

First Round Observations Gesture discovery problem

First Round Observations Gesture discovery problem Context preservation & occlusion problem

First Round Observations Gesture discovery problem Context preservation & occlusion problem

First Round Observations Gesture discovery problem Context preservation & occlusion problem Recognition problem for

First Round Observations Gesture discovery problem Context preservation & occlusion problem Recognition problem for out-of-vocabulary word UIST ? List? , Dist? , Gist? , Mist? , Hist?

Second Round Observations Highlighter hints Smart positioning of search panel Personalized indexer

Second Round Observations Highlighter hints Smart positioning of search panel Personalized indexer

Summary In situ search experience Optimum workflow / maximum flexibility Enable rich tradeoffs Gather

Summary In situ search experience Optimum workflow / maximum flexibility Enable rich tradeoffs Gather content Minimize search screen real estate Span application boundaries Tailored to pen input

Future Directions Pull in materials for searches from notes

Future Directions Pull in materials for searches from notes

Future Directions Pull in materials for searches from notes Support vector-based searches

Future Directions Pull in materials for searches from notes Support vector-based searches

Questions?

Questions?