Handling the Literature Prof Carole Goble carole goblemanchester

Handling the Literature Prof Carole Goble carole. goble@manchester. ac. uk COMP 80122 28 January 2015

You will read a lot of papers Most will be irrelevant Many will be poor Some will be important A few will change your life You will find lots of papers or none Find, Manage, Read, Synthesize ….

Purpose: Announce and Convince Defend results are plausible or correct and method convincing and repeatable. Is it “true”? Can I repeat it? Am I convinced? Is it plausible? Review & Learn Verify the results Can I reproduce it? empirically. Trust. Understand. Convince, comfort, credibility. Reuse Use the explained and Can I use it? trusted results (data, method) for Is it a useful contribution? new / my science on demand. Compare. Extend.

Scholarly Communication Forms • Making an impact – Demo, Magazine articles: reviewed – Blogs, twitter, forums: unreviewed – Technical reports • Proposing an idea or view – Position statement, Commentary, Perspectives, Magazine Department, Doctoral Consortiums – Highly cited, editorialised, low rigour, established figures • Presenting a preliminary research finding, on-going work, ideas, small extensions to existing work, • Short paper, workshop paper, poster • Medium rigour, peer review

Scholarly Communication Forms • Presenting a research finding • National Conferences: – new ideas/applications/tools, medium extensions, more serious reviewing • International Conferences: – mature work, serious reviewing, but time-constrained, check track • Journal article: – lots of mature work (e. g. , 2 conference papers into 1 journal paper), serious reviewing, not timeconstrained – High rigour, peer-review

Additional material • • • Conference papers means a presentation Slides, videos Web pages Blogs Technical reports Other?

More forms • • Position paper Systems paper Theory paper Vision paper End-to-End paper Surveys papers Summary papers … • Deep and narrow • Broad and shallow

Salami paper writing Challenge: rebuilding a body of work

Finding and choosing papers • Key players and key papers everyone cites

Finding and choosing papers • Special repositories / libraries

Lab Note books • Use one. A book. Or electronic. • A wiki? A Blog? http: //www. atriumresearch. com/html/eln. htm

Finding and choosing papers • Citation management

Managing your references • Use a reference management system http: //en. wikipedia. org/wiki/Comparison_of_reference_management_software

Places to find tools https: //www. force 11. org/catalog

Utopia Documents: http: //getutopia. com

Kristian Garzi a Ph. D student speaks about tools

I can’t find any papers! • Adjacent fields? • Different terminology? • Related topics?

Synthesize: beyond Shopping Lists • Annotated Bibliography • Literature review framework – Categories – Clusters – Timelines • Comparisons on aspects – Cross cutting the papers

Mind-mapping tools can help • http: //cmap. ihmc. us/ • http: //www. mindjet. com/uk /mindmanager/ • http: //freemind. sourceforge. net/wiki/index. php/Main_P age • http: //www. biggerplate. co m/ http: //www. biggerplate. com/mindmaps/Nw. Uu. Yp. Zx/critical-literature-review-template

Smart Reading • • • Understand the context of the paper. Beginnings and endings. Survey the structure. Use figures and tables. Generate them if absent. Decide when to read every word. Summarise the paper. Read out loud. Explain the paper to your cat. Multiple reading passes. Over time. Set aside time to think about it and digest it. Get help. Talk to people. Set up a reading group.

Where does it fit with my work • • Is it relevant? If not why not? How does it fit with your framework? Yes – you will need a framework! Can you relate the terminology and notation to yours? • Keeping a record of the contribution.

When will the paper become relevant? • Over time • An ongoing framework • Revisiting

Hints for Reviewing Papers • • The answer to each question tells you something about the technical content of the paper The ease of extracting the answer to each question tells you something about the quality of the writing. Questions • Is this a vision/position/direction paper, or a measurement/implementation paper? • If you know the area well, can you mentally slot this paper somewhere in the taxonomy? ("Differs from X as follows; has the following in common with Y; " etc. ) If the paper is radically brilliant, new, or iconoclastic work, this question may not apply. • Can you summarize the single most important contribution in one or two sentences? Issues • Will this advance the state of the art? • Did you learn anything new? • Does it provide evidence which supports/contradicts hypotheses? • Experimental validation? • How readable is the paper? • Is the paper relevant to a broader community? Goals of Review • Guide the program committee in selection process • Help authors (to revise paper for acceptance, to understand rejection, to improve further research and future projects) John Ousterhout's Hints for Reviewing Papers

Make yourself a template • • • Author housekeeping stuff Paper genre Problem statement/motivation Key ideas Technical contribution Technical flaws Evaluation Presentation Comparison • • • How would I extend this paper? What questions does it raise? Future work of author. What else? Author log – To authors’ other work – To third party’s work – To your work

Make yourself a template • • • Author housekeeping stuff Paper genre Problem statement/motivation Key ideas Technical contribution Technical flaws Evaluation Presentation Comparison – To authors’ other work – To third party’s work – To your work • • • Well-established class of problems, e. g. , FO theorem proving, image retrieval etc. Novel class of problems (is it really new? ) Single problem or many problems How would I extend this paper? Implicit or explicit new way of thinking about a problem? What questions does it raise? Future work of author. What else? Author log

Make yourself a template • • • • Author housekeeping stuff Paper genre Problem statement/motivation Key ideas Technical contribution Technical flaws Evaluation Presentation Comparison – To authors’ other work – To third party’s work – To your work How would I extend this paper? What questions does it raise? Future work of author. What else? Author log Implicit or explicit new way of doing things? New, i. e. , developed by authors? existing? Developed by authors or others? New combination of existing techniques? Good or better/worse than X and why

Make yourself a template • • • Author housekeeping stuff Paper genre Problem statement/motivation Key ideas Technical contribution Technical flaws Evaluation Presentation Comparison • • • How would I extend this paper? What questions does it raise? Future work of author. What else? Author log – To authors’ other work – To third party’s work – To your work What is the author's thesis? What are they trying to convince you of? Summarize the author's argument. How does the author go about trying to convince you of thesis?

Make yourself a template • • • Author housekeeping stuff Paper genre Problem statement/motivation Key ideas Technical contribution Technical flaws Evaluation Presentation Comparison – To authors’ other work – To third party’s work – To your work • • • How would I extend this paper? What questions does it raise? Future work of author. What else? Author log Does the author describe other work in the field? If so, how does the research described in the paper differ from the other work?

Make yourself a template • • • • Author housekeeping stuff Paper genre Problem statement/motivation Key ideas Technical contribution Technical flaws Evaluation Presentation Comparison – To authors’ other work – To third party’s work – To your work How would I extend this paper? What questions does it raise? Future work of author. What else? Author log Empirical (run tests): test suite and testing must match problem targeted Theoretical: correct and understandable/convincing and relevant Does the paper succeed? Are you convinced of thesis by the time that you have finished reading the paper?

Make yourself a template • • • Author housekeeping stuff Paper genre Problem statement/motivation Key ideas Technical contribution Technical flaws Evaluation Presentation Comparison • • • How would I extend this paper? What questions does it raise? Future work of author. What else? Author log – To authors’ other work – To third party’s work – To your work Does the author indicate how the work should be followed up on? Does the paper generate new ideas?

What are your tips? Reading groups Printing and physical markup Not printing and electronic markup Read everything 3 times