Systematic Literature Review Problem Statement LECTURE 4 UNIM

Systematic Literature Review & Problem Statement LECTURE 4 UNIM 523 – RESEARCH METHODOLOGY

Systematic Literature Review Aim is to reduce uncertainty by a rigorous methodology that is: ◦ ◦ Comprehensive Transparent or explicit Leads to minimum bias Reproducible

Key Characteristics of Systematic Reviews Clearly stated title and objectives Comprehensive strategy to search for relevant studies (unpublished and published) Explicit and justified criteria for the inclusion or exclusion of any study Clear presentation of characteristics of each study included an analysis of methodological quality Comprehensive list of all studies excluded and justification for exclusion Linda N. Meurer, MD, MPH Department of Family and Community Medicine. “Systematic Synthesis of the Literature: Introduction to Meta-analysis”.

Characteristics of Systematic Reviews (cont. ) Clear analysis of the results of the eligible studies ◦ statistical synthesis of data (meta-analysis) if appropriate and possible; ◦ or qualitative synthesis Structured report of the review clearly stating the aims, describing the methods and materials and reporting the results Linda N. Meurer, MD, MPH Department of Family and Community Medicine. “Systematic Synthesis of the Literature: Introduction to Meta-analysis”. Power Point Presentation.

An author of a good Systematic Review… Formulates a Question Conducts a Literature Search Refines the search by applying predetermined inclusion and exclusion criteria Extracts the appropriate data and assess their quality and validity Synthesizes, interprets, and reports data

Inclusion/Exclusion Criteria “Once the study question is formalized, the authors must compose a comprehensive list of inclusion and exclusion criteria. ” “To avoid selection bias, inclusion and exclusion criteria should be agreed upon and formalized before data extraction and analysis. ” Margaliot, Zvi, Kevin C. Chung. “Systematic Reviews: A Primer for Plastic Surgery Research. ” PRS Journal. 120/7 (2007) p. 1836

Literature Search “A comprehensive and reproducible literature search is the foundation of a systematic review. ” Margaliot, Zvi, Kevin C. Chung. “Systematic Reviews: A Primer for Plastic Surgery Research. ” PRS Journal. 120/7 (2007) p. 1837

Literature Search Challenges Database Bias - “No single database is likely to contain all published studies on a given subject. ” Publication Bias - selective publication of articles that show positive treatment of effects and statistical significance. ◦ Hence, it is important to search for unpublished studies through a manual search of conference proceedings, correspondence with experts, and a search of clinical trials registries. Margaliot, Zvi, Kevin C. Chung. “Systematic Reviews: A Primer for Plastic Surgery Research. ” PRS Journal. 120/7 (2007) p. 1837

Literature Review Challenges (cont. ) English-language bias - occurs when reviewers exclude papers published in languages other than English Citation bias - occurs when studies with significant or positive results are referenced in other publications, compared with studies with inconclusive or negative findings

Searching for representative literature Which organisations? Which key opinions? Which databases? Which web sites?

Formulate Research Q & Develop protocol Clear research question MUST be formulated to guide the SR Develop a detail, step by step protocol ◦Refined with experts ◦Update along the way 11

OUTLINE OF THE RESEARCH PROTOCOL • • • TITLE OF THE RESEARCH PROJECT RESEARCH SUMMARY BACKGROUND REVIEW QUESTION SEARCH STRATEGY METHODS OF STUDY SELECTION QUALITY ASSESSMENT DATA EXTRACTION DATA SYNTHESIS BIBLIOGRAPHIC REFERENCES TIMETABLE ANNEXES (Search strategy, data extraction form) 12

Search strategy Well formulated q aid in devising the best search strategy: ◦ Break down Qs into parts/concepts ◦ Gather terms for each concept by: ◦ Synonyms ◦ Abbreviations ◦ Related terms ◦ Transatlantic differences ◦ Identify main topic areas 13

An overview of the Literature Search strategy 14

15 Search strategy Define each concept in the research topic clearly Define inclusion and exclusion criteria Define specific type of study for filtering Formulate individual search terms for each DB + filters Limit to subject, language and year Iterative process ◦ Modify previously used terms based on retrieval ◦ Check search results for relevancy and numbers

Search strategy Databases Terms medical, informatics, engineering (health OR medical) informatics, system, computer based, (evaluation OR assessment) methodical, empirical assessment of HIS in clinical settings Evaluation study Study design Case study HIS CBIS that involves human interaction Limit human subjects, English language, 1985 -2005 (Ammenwerth and de Keizer, 2005) 16

17 Study selection methods Electronic and manual searches Save/import all citations into a folder, labeled according to sources Transfer citations according to screening level Log your search: study inclusion and exclusion (based on research q) Exclusion: ◦ Duplicate ◦ Study type ◦ Lack/no outcome ◦ Non article based

Sources of electronic and hand searches Sources Databases The Cochrane Library (http: //www. cochrane. org) The Cochrane Database of Systematic Reviews The Database of Abstracts of Reviews of Effects The Cochrane Controlled Trials Register The Cochrane Methodology Register The NHS Economic Evaluation Database Health Technology Assessment Database Cochrane Database of Methodology Reviews (CDMR) Major subject focuses databases Pub. Med MEDLINE EMBASE, ISI Web of Knowledge, Ingenta ACM, IEEE, Springer, Scopus Researchers Personal contact with experts in the health informatics field (J. Wyatt and E. Ammenwerth) Specialist databases Health Management Information Consortium (HMIC) National Electronic Library for Health (Ne. LH). http: //www. nelh. nhs. uk/ Organising Medical Networked Information (OMNI) (http: //www. omni. ac. uk) Ovid (http: //www. ovid. com/ Web searching of research registers Consulting registers of ongoing and recently completed research Search engines: Google, Yahoo – this search method was conducted before the conduct of this systematic review. Thus, the actual number of search results prompted was not recorded. National Research Register (NRR) reports. Records research projects being undertaken in the NHS and of interest to the NHS. http: //www. updatesoftware. com/National/ Dissertation Abstracts (http: //wwwlib. umi. com/dissertations/ Networked Digital Library of theses and dissertations (NDLTD) via Scirus search engine http: //www. ndltd. org/ Journals selected for hand search Journals were selected for hand search among those that were found to have the highest yield of studies on standard bibliographic searching. Journal of the American Medical Informatics Association Medical Decision Making Methods in Information and Medicine International Journal Of Medical Informatics Artificial Intelligence Medicine Journal Of Evaluation In Clinical Practice Journal of American Medical Association (JAMA) British Medical Journal (BMJ) Citation searching and chaining 9 studies listed in the reference list of an article were traced. 18

Methods of study selection Total references retrieved and searched N = 12074 Total abstracts screened N = 11108 Exclude duplicate N=966 Rejected at abstract N=10 441 Total full texts screened N = 698 Rejected full texts N=639 Total included papers N = 59 Final included papers N = 55 Rejected study quality N=4 19

Initially identified 4802 *** Title/abstract scanned Included 173 Full report 102 Full paper scanned Non grey 12 Grey 8 20

Different type of grey literature Annual reports Theses/dissertations Conference abstracts/proceedings Working papers Evaluation reports Facts sheets and others

Advantages of including ‘grey’ in a literature review Access to wider variety of information Reduces publication bias Contain more local information- important for planning Contain process information- important for policy makers

Advantages of including ‘grey’ in a literature review Reports not published in peer reviewed journals (e. g. UN, WHO, HTA) Access to new research World Wide Web (4 -6 Billion web pages) Contain valuable and unique information, which is not found elsewhere

Quality Assessment Questions (Greenhalgh and Taylor, 1997) Are the aims and objectives clearly stated? Is the research design clearly specified and appropriate for the aims and objectives? Are study methods such as data collection clearly defined? How were the setting and the subjects selected? Is the analytic method appropriate and adequately explained? Are the results credible, and if so, are they clinically important? Cite actual data/ verbatim quotes What conclusions were drawn, and are they justified by the results? 24

Data extraction Design a proforma to capture general info, study objectives/q, method, findings and theme Based on research question Pilot test on 10 papers 25

Data analysis Quantitative, qualitative or both Mix of textual commentary and summary tables ◦ Important studies characteristics ◦ Important studies’ results and conclusions contributing to the review 26

E. g. of Textual Commentary

Results: Study Descriptions Country largely from the USA (56. 4%); Canada(10. 9%); Australia(9. 1%); UK(7. 3%) Participants Managers, IT people, clinicians, clerical staff (3 -1500) Methods Mix +-50% Eval. focus Eval. time H-O-T +-40% Summative N=44 (80%) Formative N= 2 S + F N =9 29

Type of HIS 30

Dominant factors of HIS adoption: The top 10 Adoption factors Leadership & support Ease of use Info. accessibility Category O T T N= 34 28 26 System usefulness User training System time (response & turnaround) Clinical process Info. relevancy System flexibility User perception T H T O T T H 24 24 23 20 20 18 18 31

Dominant factors of HIS adoption T SQ: ease of use, usefulness, time, flexibility, reliability, technical support SD: user involvement, system purpose, installation strategy SU: training, H perception, skills attitude role IQ: accessibility, relevancy, completeness, legibility, format communication OS: Leadership & support, clinical process, internal comm. , resources, org. readiness O OE: IOS 32

Practical issues Protocol is very useful in conducting the SR ◦ Adhere and update it Use of reference manager ease and speed up the SR Difficulty and time consuming process of obtaining full paper 33

Practical issues: papers problems Lack of methodology description resolutions • Consider findings importance to theory and practice. • Check source credibility • Contact authors Studies heterogeneity (terms • Refer to established or selfvariety and ambiguity, defined taxonomy or definitions overlapping/redundant/ • Extract data from a series of replicated findings, reporting publications from the same styles) study together 34

Example: The use of Social Media In Communicating and Creating Environmental Sustainability Awareness in Malaysian Higher Learning Institutions 1. Searched for articles (journals and conference papers) from three databases: ISI Web of Science; Scopus; and Science. Direct which basically are sufficient to cover all past and recent research on the topic. KEYWORD 01 (KW 1): “environmental awareness” AND “higher education” [AND “university”] Keyword (KW) KW 1 KW 2 KW 3 KW 4 Total ISI Web of Science 13 0 4 2 19 Scopus 28 0 6 1 35 Science Direct 343 6 289 249 887 Total 384 6 299 252 941 KEYWORD 02 (KW 2): “social media” AND “environmental awareness” AND “higher education” [“social media” was also replaced with “social technology”, “social technologies”, “Facebook”, “Twitter”, and “blog” as to include extensiveness) KEYWORD 03 (KW 3): “factors” AND “environmental awareness” AND “higher education” KEYWORD 04 (KW 4): “benefits” AND “environmental awareness” AND “higher education”

Step 2 Filtering the articles such that only relevant papers from the database are considered for this research. Filtering process undergoes 3 cycles. Filtering Cycle #1: a) b) c) d) Removing the duplication by looking one by one of the 941* citations from each of the keywords (KW 1 -KW 4) and databases (Web of Science, Scopus and Science. Direct) For (a) above, using End. Note is the best way. Using End. Note, also remove the “peculiarity” such as articles with missing author, year, title and too old articles (those published before 2000 needs to be removed. However, be careful because some databases did not capture Year information at all in the RIS file) � this is the basic inclusion and exclusion criteria After removing the duplicate citations, we got a new number: “ 705” articles.

Step 2: Filtering Cycle #2 Using the remaining “ 705” articles from Cycle #1, then we go through the “Title” of the papers (from the title, we will know the context of the paper. If not environmental awareness in higher education, then we will remove it) Then we decide whether the paper is relevant to the research or not; If relevant, we keep it, else we remove it (i. e this is the application of inclusion and exclusion criteria) At the end of Cycle #2, we got a new number: “ 81” articles. Proceed to downloaded the “ 82” number of articles from the databases.

Step 2: Filtering Cycle #3 Using the remaining “ 82” articles from Cycle #2, then we go through the “abstract” and “detail contents” of the papers. Then we decide whether the paper is REALLY RELEVANT & USEFUL TO INFORM this research or not; If relevant, we keep it, else we remove it (i. e this is also the application of inclusion and exclusion criteria) At the end of Cycle #3, we will either get a new number: “X” (i. e less than 82) articles or maintain using the 82 articles.

Step 3: Detail analysis Using the finalised “ 82” (or less i. e “X”) from Cycle #3, we will analyse the papers systematically as to look at: ◦ The concept of environmental awareness being discussed ◦ The contexts (i. e type of higher education: university, college; country: China, Turkey, Singapore, Malaysia, etc; focus: students, lecturers, administrators). If context is not higher education, we will remove it. ◦ The use of social media (i. e Facebook, Twitter, blog, etc) or non-social media (i. e email, website, circular, etc) ◦ How the social media being used to create and communicate environmental awareness ◦ The theory used to discuss the findings ◦ The methodology to conduct the research The findings of the papers

Concept of Env. Awareness Paper 01 (replace this later with paper detail: authors, year, title) (isi dlm column ni sample je) Paper 02 (sample je ni) Contexts (HE/Country) Types of Social Media Used Env awareness is discussed from the aspect of encouraging recycling in the university China. University None in Beijing. mentioned Students and staff. Env awareness is about building the culture of green among young people. Malaysia. College in KL. Students, staff, city council. Roles of Social Media None mentioned. Theory Used in the Research TAM Research Methodology Quantitative. Survey of 200 students. Key Findings 1. 2. 3. Facebook Create Activity Theory awareness via educational programme and inclusion in academic curriculum Qualitative. 1. Semi-structured interviews of 30 people. Case studies. 2. Other University set green policy Leadership by top management Case study of recycling Culture of green is encouraged via constant communication College implemented ISO 14001 and Earth Day involving staff and students A model of green culture is developed (refer page XYZ)

Step 4: Analyse – Produce Themes/Identify construct From the literature extraction - finalize the task by compiling the extraction and coming out with broad themes. Compile the inputs and combined them into a table to further analyse them. The initial analysis were then further analysed to generate broad themes. We call them finalised theming. After that, provide conclusion to the data analysis

Step 5: Conclusion from the analysis For the SLR, we used nine (9) elements to analyse the 82 previous research articles on this topic: 1. The concept of environmental sustainability (ES) 2. The context of ES 3. The types of social media in ES awareness in HE 4. The roles of social media in ES awareness sin HE 5. The focus parameters used by previous research on ES awareness in HE 6. The theories used in previous research on ES awareness in HE 7. The research methodologies used in previous research on ES awareness in HE 8. The key findings from previous research on ES awareness in HE 9. The limitations and future works as suggested by previous research on ES awareness in HE

Derive RQs Based on Kitchenham, et. al (2009) Based on Kitchenham, B. , Pearl Brereton, O. , Budgen, D. , Turner, M. , Bailey, J. , & Linkman, S. (2009). Systematic literature reviews in software engineering–a systematic literature review. Information and software technology, 51(1), 7 -15. SLR paper by Kitchenham, et. al (2009) is based on discussion on research questions (RQs). Hence, as to comply with the actual SLR tradition, the nine elements were grouped based on their relevance and logic, then converted into 6 research questions, as follow: Research Question 1: How is environmental sustainability awareness viewed in higher education? (Element 1) Research Question 2: What is the context of environmental sustainability discussion in higher education? (Element 2) Research Question 3: What are the types and roles of social media used in creating environmental sustainability awareness in higher education? (Element 3 + Element 4) Research Question 4: What are the focus parameters in researching environmental sustainability awareness in higher education? (Element 5) Research Question 5: What are theories and research methodologies used in researching environmental sustainability in higher education? (Element 6 + Element 7) Research Question 6: What are the key findings, limitations and future research works on environmental sustainability awareness? (Element 8 + Element 9)

RQ 2: What is the context of environmental sustainability discussion in higher education? 1. 2. 3. HIGHER EDUCATION CONTINENTS OF WHERE THE ARTICLE IS BASING ON TYPES OF ENV. EDUCATION Summary of findings for RQ 2: This means that environmental sustainability and its awareness is an area of interest to researchers in many higher educations across the globe; and in various academic programs for it to be embedded. RQ 3: What are the types and 1. 2. SOCIAL MEDIA NON-SOCIAL MEDIA roles of social media used in 1. DISSEMINATE AWARENESS creating environmental 2. DELIVERY OF ENV EDUCATION sustainability awareness in Summary of findings for RQ 3: higher education? This means that environmental sustainability in higher education are not widely using social media or those using social media are not researched and published in the extant literature. Further, there are two roles of social media is to disseminate information and education about environmental awareness. While non-social media was used to deliver teaching and learning related to environmental education.

RQ 4: What are the focus parameters in 1. researching environmental sustainability awareness in higher education? 1. Environment awareness 2. Environmental education 3. Environmental knowledge 4. Env & Kn dissemination 5. Environmental attitude 6. Environmental practices 7. Environmental engagement 8. Environmental leadership 9. Environmental funding 10. Environmental vision 11. Environmental issues 12. Environmental benefits Factors a. Internal environmental factors b. Social factor c. Economic factor d. Environmental factor e. Technical factor f. Political factor Summary of findings for rq 4: This means that environmental sustainability is researched from various aspects or angles. Towards creating environmental sustainability, environmental education to build environmental knowledge and ways to disseminate the environmental knowledge are discussed. Environmental awareness also deals with people’s environmental attitudes, environmental practices of individuals or organisations, and the environmental engagement they received. For environmental awareness to be successful, environmental leadership is important. Whereby, funding needs to be allocated, visions set and implemented, issues addressed and benefits acknowledged and propagated. For environmental awareness to build, several factors, either internal or external are also in play. These include internal environmental factors, and external ones such as social, economical, political, technical, and also environmental.

Conclusions SR is a challenging and tedious but satisfying and rewarding process 46

PROBLEM STATEMENT The HEART of a doctoral dissertation IS the PROBLEM STATEMENT. This is the place where most faculty members go first to understand assess the merits of a proposal or a dissertation. After reading the problem statement, the reader will know why you are doing this study and be convinced of its importance/novelty.

PROBLEM STATEMENT Steps to be Taken: • Identify a general problem area • Select a real problem • Revise & narrow the problem statement

PROBLEM STATEMENT… Identify a general problem area • • • Interest Theoretical value Practical value Workability Critical mass

PROBLEM STATEMENT… Identify the real problem… • • • Know thoroughly the body of research and the techniques related to the chosen methodology suite to the topic chosen Review the literature - Get broad overview using secondary sources Write a general problem statement

PROBLEM STATEMENT… Identify a general problem area… • Review the literature - Get broad overview using secondary sources • Write a general problem statement • Outline the Problem - Topic - Statement of the problem

PROBLEM STATEMENT… Writing of problem statement… • Tells what will be (or was) done • Identifies variables & relationships to be studied • REVISE & NARROW THE PROBLEM STATEMENT • Review primary sources in the literature • Make a list of research questions • Provide a rationale for the problem • Identify variables to be studied • REWRITE THE PROBLEM STATEMENT

PROBLEM STATEMENT… In 200 words or less (about 1 -2 paragraphs) you need to convince the reader that this study MUST be done!

Example PROBLEM STATEMENT… 1: Integrating Emotional Intelligence in E-Learning Systems to Reduce Attrition Rate Emotional intelligence includes the ability to be aware and in control of one’s own emotions, to be empathic with others, to motivate oneself, and to be effective in intrapersonal and interpersonal relationships (Cooper & Sawaf, 1997; Gardner 1993, 1999; Goleman 1995, 1998; Mayer & Salovey 1997; Stein & Book, 2000; Weisinger 1998). When human communication is involved, emotional intelligence is involved (Rivera, 2000 a, 2000 b). E-Learning has been introduced since the 1990 s. It has been an effective form of educational communication at all levels. However, research has shown that e-leaning have not been able to sustain learners long in the programmes and there is perhaps a correlation between emotional intelligence and attrition rate in e-learning.

PROBLEM STATEMENT… Example 2 Attrition appears to be a serious problem in E-learning, as the dropout rate is more than twice that of traditional forms of learning (Flood, 2002; Lynch, 2001). Learning more about the relationship between emotional intelligence & E-learning and developing a specially designed system might not only help offer an explanation for the unacceptably high attrition rate of learners in E-Learning programmes but could also help reduce the attrition rate. Therefore, this study will develop an e-learning model that integrates Emotional Intelligence and provide activities that will encourage Emotional Intelligence characteristics through specialised activities that would reduce the attrition in e-learning programmes.

Assignment #2: Due in 2 weeks : 9 July 2014