Master of System Engineering Faculty of Engineering Gadjah

Master of System Engineering Faculty of Engineering Gadjah Mada University Research Methodology and Project Proposal Preparation Adhy Kurniawan Faculty of Engineering Gadjah Mada University

Adhy Kurniawan ► 1987 -1990 SMA 3 Semarang ► 1990 -1991 Fac. Of Economy, Diponegoro University, Semarang ► S 1(1991 -1996) Civil Engineering Dept. Gadjah Mada Univ. ► S 3(1998 -2003) Swiss Federal Institute of Technology Lausanne (EPFL), Swiss ► Post Doct (nov. 2005 -sept. 2006) Kyoto University, Japan

My Goals for Course ►That each of you develop an intuition for the fundamental principles of research methodology ►That we have an enjoyable semester learning together

Lecture and Homework Lecture • Presentation and discussion Homework ►Your chance to practice using the concepts presented in class ►Teamwork vs. Individual work?

References ►All of literature concerning: Res Met ► Marczyk, De. Matteo, Festinger. 2005, Essentials of Research Design and Methodology, John Wiley and Sons. ►Day and Gastel, 2006, How to write and Publish a Scientific Report, Greenwood Press ►Metodologi Riset, Etc.

List of students ► ► ► ► ► Alif Ardy Saputra, Geodesi UGM Anik FR, TL, ITB Ashri Uswatun, TFisika, UGM Ayi Fajarwati, TL, ITB Corry Agustina, Perenc Wil, TA, UGM Dwi Astuti, TKimia, UGM Elva Nur , TF, UGM Erika Kezia, TL, ITB Fitri Wijayanti, Fisika, UNS I Nyoman Kusuma, TF, UGM Ihsan Hasan, T Industri, UII, Ihwan Ghazali, T Industri, UAD Iin Lestari, TL, ITB M Sony Abertiawan, TL, ITB Maria Auliana, T Sipil, UGM Norma Pradipta, TArsitektur, UGM Satrya Alrizki, TGeofisik, ITB Tatag Lindu Bhakti, TFisika, UGM

Contents The aims of research, the research topic, title and research problem, literature review, research design: population and sampling types, types of quantitative research designs, validity of conclusions, data-collecting methods and measuring instruments in quantitative research, qualitative research designs, data analysis and interpretation of results, report writing and the research proposal, ethical consideration on research.

OVERVIEW OF SCIENCE AND THE SCIENTIFIC METHOD science can be defined as a methodological and systematic approach to the acquisition of new knowledge. This definition of science highlights some of the key differences between how scientists and nonscientists go about acquiring new knowledge. Specifically, rather than relying on mere casual observations and an informal approach to learn about the world, scientists attempt to gain new knowledge by making careful observations and using systematic, controlled, and methodical approaches (Shaughnessy & Zechmeister, 1997). Shaughnessy, J. J. , & Zechmeister, E. B. (1997). Research methods in psychology (4 th ed. ). Boston: Mc. Graw Hill.

►In addition, scientific knowledge is not based on the opinions, feelings, or intuition of the scientist. ►Instead, scientific knowledge is based on objective data that were reliably obtained in the context of a carefully designed research study. ►In short, scientific knowledge is based on the accumulation of empirical evidence (Kazdin, 2003 a) Kazdin, A. E. (2003 a). Methodology: What it is and why it is so important. In A. E. Kazdin ( Ed. ), Methodological issues and strategies in clinical research (3 rd ed. , pp. 5– 22). Washington, DC: American Psychological Association.

►The defining characteristic of scientific research is the scientific method. ►First described by the English philosopher and scientist Roger Bacon in the 13 th century, it is still generally agreed that the scientific method is the basis for all scientific investigation. ►The scientific method is best thought of as an approach to the acquisition of new knowledge, and this approach effectively distinguishes science from nonscience.

The Scientific Method The development of the scientific method is usually credited to Roger Bacon, a philosopher and scientist from 13 th-century England, although some argue that the Italian scientist Galileo Galilei played an important role in formulating the scientific method. Later contributions to the scientific method were made by the philosophers Francis Bacon and René Descartes.

►Although some disagreement exists regarding the exact characteristics of the scientific method, most agree that it is characterized by the following elements: ● ● ● ● Empirical approach Observations Questions Hypotheses Experiments Analyses Conclusions Replication

Empirical Approach ► The scientific method is firmly based on the empirical approach. The empirical approach is an evidence-based approach that relies on direct observation and experimentation in the acquisition of new knowledge (see Kazdin, 2003 a). ► In the empirical approach, scientific decisions are made based on the data derived from direct observation and experimentation. ► Contrast this approach to decision making with the way that most nonscientific decisions are made in our daily lives. ► For example, we have all made decisions based on feelings, hunches, or “gut” instinct. Additionally, we may often reach conclusions or make decisions that are not necessarily based on data, but rather on opinions, speculation, and a hope for the best. ► The empirical approach, with its emphasis on direct, systematic, and careful observation, is best thought of as the guiding principle behind all research conducted in accordance with the scientific method.

Observations ► An important component in any scientific investigation is observation. In this sense, observation refers to two distinct concepts—being aware of the world around us and making careful measurements. ► Observations of the world around us often give rise to the questions that are addressed through scientific research. ► For example, the Newtonian observation that apples fall from trees stimulated much research into the effects of gravity. Therefore, a keen eye to your surroundings can often provide you with many ideas for research studies.

Questions ► After getting a research idea, perhaps from making observations of the world around us, the next step in the research process involves translating that research idea into an answerable question. ► The term “answerable” is particularly important in this respect, and it should not be overlooked. ► It would obviously be a frustrating and ultimately unrewarding endeavor to attempt to answer an unanswerable research question through scientific investigation. ► It is therefore important to formulate a research question that can be answered through available scientific methods and procedures.

Hypotheses ► The next step in the scientific method is coming up with a hypothesis, which is simply an educated—and testable —guess about the answer to your research question. ► A hypothesis is often described as an attempt by the researcher to explain the phenomenon of interest. ► Hypotheses can take various forms, depending on the question being asked and the type of study being conducted. ► A key feature of all hypotheses is that each must make a prediction. ► Remember that hypotheses are the researcher’s attempt to explain the phenomenon being studied, and that explanation should involve a prediction about the variables being studied. ► These predictions are then tested by gathering and analyzing data, and the hypotheses can either be supported or refuted on the basis of the data.

► Two types of hypotheses with which you should be familiar are ● the null hypothesis ● and the alternate (or experimental) hypothesis. ► The null hypothesis always predicts that there will be no differences between the groups being studied. ► By contrast, the alternate hypothesis predicts that there will be a difference between the groups. ● For example, ● the null hypothesis would predict that the exercise group and the no-exercise group will not differ significantly on levels of cholesterol. ● The alternate hypothesis would predict that the two groups will differ significantly on cholesterol levels. ● Homework: Individual ● Please try to find one example. About the null

Experiments ► After articulating the hypothesis, the next step involves actually conducting the experiment (or research study). ► For example, if the study involves investigating the effects of exercise on levels of cholesterol, the researcher would design and conduct a study that would attempt to address that question. ► As previously mentioned, a key aspect of conducting a research study is measuring the phenomenon of interest in an accurate and reliable manner. ► In this example, the researcher would collect data on the cholesterol levels of the study participants by using an accurate and reliable measurement device. ► Then, the researcher would compare the cholesterol levels of the two groups to see if exercise had any effects.

Accuracy vs. Reliability ► When talking about measurement in the context of research, there is an important distinction between being accurate and being reliable. ► Accuracy refers to whether the measurement is correct, whereas reliability refers to whether the measurement is consistent. ► An example may help to clarify the distinction. ● When throwing darts at a dart board, “accuracy” refers to whether the darts are hitting the bull’s eye (an accurate dart thrower will throw darts that hit the bull’s eye). ● “Reliability, ” on the other hand, refers to whether the darts are hitting the same spot (a reliable dart thrower will throw darts that hit the same spot). ● Therefore, an accurate and reliable dart thrower will consistently throw the darts in the bull’s eye. As may be evident, however, it is possible for the dart thrower to be reliable, but not accurate. ● For example, the dart thrower may throw all of the darts in the same spot (which demonstrates high reliability), but that spot may not be the bull’s eye (which demonstrates low accuracy).

Analyses ► After conducting the study and gathering the data, the next step involves analyzing the data, which generally calls for the use of statistical techniques. ► The type of statistical techniques used by a researcher depends on the design of the study, the type of data being gathered, and the questions being asked. ► It is important to be aware of the role of statistics in conducting a research study. ► In short, statistics help researchers minimize the likelihood of reaching an erroneous conclusion about the relationship between the variables being studied.

Conclusions ► After analyzing the data and determining whether to reject the null hypothesis, the researcher is now in a position to draw some conclusions about the results of the study. ► For example, if the researcher rejected the null hypothesis, the researcher can conclude that the phenomenon being studied had an effect—a statistically significant effect, to be more precise. ● If the researcher rejects the null hypothesis in our exercise-cholesterol example, the researcher is concluding that exercise had an effect on levels of cholesterol.

►It is important that researchers make only those conclusions that can be supported by the data analyses. ►Going beyond the data is a cardinal sin that researchers must be careful to avoid.

Replication ►One of the most important elements of the scientific method is replication. ►Replication essentially means conducting the same research study a second time with another group of participants to see whether the same results are obtained. ►The same researcher may attempt to replicate previously obtained results, or perhaps other researchers may undertake that task.

► Replication illustrates an important point about scientific research—namely, that researchers should avoid drawing broad conclusions based on the results of a single research study because it is always possible that the results of that particular study were an aberration. ► In other words, it is possible that the results of the research study were obtained by chance or error and, therefore, that the results may not accurately represent the actual state of things. ► However, if the results of a research study are obtained a second time (i. e. , replicated), the likelihood that the original study’s findings were obtained by chance or error is greatly reduced.

►What are three general goals of scientific research?

Answer: ►description, ►prediction, ►and understanding/explaining

What Exactly is Research? ►we will focus on two of the most common types of research— ● correlational research ● and experimental research

Correlational research: ► In correlational research, the goal is to determine whether two or more variables are related. (By the way, variables” is a term with which you should be familiar. ► A variable is anything that can take on different values, such as weight, time, and height. ) ► For example, a researcher may be interested in determining whether age is related to weight. In this example, a researcher may discover that age is indeed related to weight because as age increases, weight also increases. If a correlation between two variables is strong enough, knowing about one variable allows a researcher to make a prediction about the other variable. It is important to point out, however, that a correlation— or relationship—between two things does not necessarily mean that one thing caused the other. To draw a causeand-effect conclusion, researchers must use experimental research. .

Experimental research: ► In its simplest form, experimental research involves comparing two groups on one outcome measure to test some hypothesis regarding causation. ► For example, if a researcher is interested in the effects of a new medication on headaches, the researcher would randomly divide a group of people with headaches into two groups. ► One of the groups, the experimental group, would receive the new medication being tested. ► The other group, the control group, would receive a placebo medication (i. e. , a medication containing a harmless substance, such as sugar, that has no physiological effects).

Experimental research: Besides receiving the different medications, the groups would be treated exactly the same so that the research could isolate the effects of the medications. After receiving the medications, both groups would be compared to see whether people in the experimental group had fewer headaches than people in the control group. Assuming this study was properly designed (and properly designed studies will be discussed in detail in later chapters), if people in the experimental group had fewer headaches than people in the control group, the researcher could conclude that the new medication reduces headaches.

Task ►Compose your own brief research proposal. ►Try to determine your research topic for MST final project ►Format: ● 1. In MS Word ● 2. In Power point

Task/assignment next week ►Review 1 International Publication (Journal, Conference paper, etc) related to Renewable energy ►Compose the summarize of your review ►Format: ● 1. In MS Word ● 2. In Power point

Purpose of the research proposal ► 1. To inform the reader of nature of your proposed research. ● What is the problem? ● What is its extent? ► 2. To convince the reader, especially supervisors and reviewers, of the value of your proposed research. ● Is this project worth the time and money? ● Will it make a difference to the world?

Purpose of the research proposal ► 3. To demonstrate your expertise and competency in a particular area of study. ● Do you have the qualifications to conduct this research? ● Have you informed yourself of the existing theory and data relevant to your topic? ● Do you have the necessary skills to conduct the research?

Purpose of the research proposal ► 4. To plan the research project and provide a step-by-step guide to the tasks necessary for its completion. ● What are the key stages of the work? ● What are the priorities? ● How do the various components fit together? ► 5. To request support from individuals and agencies who provide supervision, oversight or funding for the research project. ● What kinds of support does the project need? ● Are all participants properly protected?

Purpose of the research proposal ► 6. To contract with the agencies and individuals involved, including supervisors, foundations and participants in the research team. ● How will tasks be assigned and resources expended? ● What does each contribute to the collective endeavor?

First things first 1. 2. 3. 4. 5. Basics Topic ideas Typical methodologies Common pitfalls Getting started and putting it all together 6. Questions/discussion

Basic steps of a research project ►Find a topic What, When ►Formulate questions What, Why ►Define population Who, When ►Select design & measurement How ►Gather evidence How ►Interpret evidence Why ►Tell about what you did and found out

Selecting a Research Topic ►What are some considerations when selecting a research topic?

Considerations in Selecting a Topic ● Personal interest / Passion ● Importance / Contribution to the field ● Newness / Relevance ● Feasibility § § § Tradeoff between rigor and practicality Time constraints Ethical constraints Organizational support Economic factors Availability of Subjects

Sources of Research Topics ● Peer-reviewed journals in your field ● Personal experiences ● Work setting experiences ● Existing literature § “Recommendations for future research…”

Refining Your Topic ►Refinement needed for effective and efficient research ● Narrow your topic ● Identify a theoretical framework ● Specifically and unambiguously define terms ● State research questions and hypotheses

Refining Your Topic (cont’d) ►A literature review will help you ● See if your idea has been tried ● Include all relevant constructs ● Select instruments ● Anticipate common problems

Components of a Concept Paper ● Title page ● Introduction ● Nature of the Problem ● Background and Significance of the Problem ● Preliminary Literature Review ● Initial Research Question or Questions

Components of a Concept Paper (cont’d) ● Brief Description of Methodology and Research Design ● Anticipated Outcomes ● Timeline ● References

The Literature Review

What is a Literature Review? • According to Creswell (2005), a review of the literature “is a written summary of journal articles, books and other documents that describes the past and current state of information, organizes the literature into topics and documents a need for a proposed study. ” (pp. 79) Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Focusing on Empirical Research ►What does Empirical Mean? ►Primary Sources ● Original Research Article ►Secondary Sources ● ● ● Newspapers Book chapters Television/Radio Magazines Wikepedia

Empirical Research ► All empirical research is inherently flawed ● Limitations 1. Sampling ● Generalizability ● Representative 2. Measurement ● Measurement Error ● Social Desirability 3. Problem Identification ● Grasping the “Whole” Problem

Literature Reviews ►Well-written analytical narrative that brings a reader up-to-date on what is known on a given topic, but also provide fresh insights that advance knowledge ● Resolve conflicts between studies ● Identify new ways to interpret research results ● Creating a path for future research

Anecdotal Reports ►A description of an event or experience that happened to be noticed ● No control ● No comparison

Review of Key Elements of Previous Definition • The LR is a summary of research: • It is not a “list” of found research but a coherent and articulate account of past and current research findings • Suggestion: read 2 or 3 LRs in order to become familiar with summary styles

Review of Key Elements of Previous Definition (cont’d) • The sources typically are journal articles, books and other documents that describe past and present status of research in a given field: • The LR should be exhaustive and as current as possible. • How many articles? • There is no set number. As long as the search is exhaustive and focused on the research topic, the review will be acceptable.

Review of Key Elements of Previous Definition (cont’d) • How far back should one search? • A reasonable and widely accepted timeframe includes research conducted during the past 10 years. Important studies (i. e. , studies that had a significant impact on the field of study) should also be mentioned even if these go beyond the mentioned timeframe.

Review of Key Elements of Previous Definition (cont’d) • The LR should be organized: • The review should not only be coherent, but should organize the studies reviewed under themes or topics. • The reviewer is a guide and should be able to provide readers with an in-depth and current status of research in a given area. • This aspect is essential for readers to understand what the reviewer found during the search.

Review of Key Elements of Previous Definition (cont’d) • The LR should document the need for a proposed study: • Studies should not duplicate research that has been already done. • Even in cases when research is duplicated (replicated is the appropriate term), one is responsible for documenting the need for replication, e. g. , need to explore the same methodology with a different group or population, or need to change methodology with the same group.

Creswell’s 5 steps to Conduct a Literature Review • Step 1: Identify Key Terms or “Descriptors” • Extract key words from your title (remember, you may decide to change the title later) • Use some of the words other authors reported in the literature Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Step 1: Identify Key Terms or “Descriptors” (cont’d) • Use the “Thesaurus of ERIC Descriptors to look for terms that match your topic: go to www. eric. ed. gov and in “Search” select “Descriptors (from Thesaurus)” • Scan both electronic and library journals from the past 10 years and look for key terms in the articles Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Creswell’s 5 steps to Conduct a Literature Review (cont’d) • Step 2: Locate Literature • Use academic libraries, do not limit your search to an electronic search of articles • Use primary and secondary sources. A “primary source” is research reported by the researcher that conducted the study. A “secondary source” is research that summarizes or reports findings that come from primary sources Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Step 2: Locate Literature (cont’d) • It is “best to report mostly primary sources” (p. 82) • Search different types of literature: summaries, encyclopedias, dictionaries and glossaries of terms, handbooks, statistical indexes, reviews and syntheses, books, journals, indexed publications, electronic sources, abstract series, and databases Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Creswell’s 5 steps to Conduct a Literature Review (cont’d) • Step 3: Critically Evaluate and Select Literature • Rely on journal articles published in national journals • Prioritize your search: first look for refereed journal articles, then, non-refereed articles, then books, then conference papers, dissertations and theses and then papers posted to websites Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Step 3: Critically Evaluate and Select Literature (cont’d) • Look for research articles and avoid as much as possible “opinion” pieces • Blend qualitative and quantitative research in your review Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Creswell’s 5 steps to Conduct a Literature Review (cont’d) • Step 4: Organize the Literature • Create a “file” or “abstract” system to keep track of what you read. Each article you read should be summarized in one page containing Title (use APA to type the title so that you can later copy-paste this into the References section of your paper) Source: journal article, book, glossary, etc. Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Step 4: Organize the Literature (cont’d) Research problem: one or two lines will suffice Research Questions or Hypotheses Data collection procedure (a description of sample characteristics can be very handy as well) Results or findings of the study • Sort these abstracts into groups of related topics or areas which can then become the different sections of your review Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Creswell’s 5 steps to Conduct a Literature Review (cont’d) • Step 5: Write a Literature Review • Types of Reviews: Thematic Review: a theme is identified and studies found under this theme are described. Major ideas and findings are reported rather than details. Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Step 5: Write a Literature Review (cont’d) Study-by-study Review: a detailed summary of each study under a broad theme is provided. Link summaries (or abstracts) using transitional sentences. Must be organized and flow coherently under various subheadings. Avoid string quotations (i. e. , lengthy chunks of text directly quoted from a source) Creswell, J. W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research

Preliminary Literature Review ►This succinct review of current literature should: ● Provide further contextual background ● Reveal issues related to your study ● Describe similar problems in other organizations ● Provide significance to your approach to the study

Guidelines on Style, Mechanics, and Language Usage ►Does your draft follow the logic or idea that is presented in your intro and title? ►Avoid overusing direct quotations, especially long ones ►Check style manual for correct use of citations ● (Doe, 2005); Doe (2005); (Doe & Smith, 2005); Doe and Smith (2005); (Black, 2005; Brown, 2006; Yellow, 2007)

Guidelines on Style, Mechanics, and Language Usage ►Avoid using synonyms for recurring words ● This is not creative writing and stay consistent with terminology § Group I, Phoenix Cohort, Experimental Group ►Spell out all acronyms when first using them ● Traditional - American Psychological Association (APA) ● Non-traditional - Collective Efficacy (CE) ►Yes - Do NOT use contractions; No – Don’t use contractions ►Coined terms should be set off by quotes

Guidelines on Style, Mechanics, and Language Usage ►Avoid the following: ● Slang – “cool” ● Colloquialisms – “thing” >> “item” or “feature” ● Idioms – “rise to the pinnacle” >> “to become prominent” ►Use great care to avoid Plagiarism

What needs to be included in the Literature review. ►Provides contextual background ►Reveals related issues ►Reviews similar problems elsewhere ►Provides significance to your approach to the study ►Includes major/seminar research articles pertaining to study ►Written in an integrated manner ►Uses peer-reviewed research ►Includes a Reference section

Writing Your Research Question(s) ►Reflect the problem that the researcher wants to investigate ►Can be formulated based on theories, past research, previous experience, or the practical need to make data-driven decisions in a work environment

Writing Your Research Question(s) (cont’d) ►Are vitally important because they, in large part, dictate what type of statistical analysis is needed, as well as what type of research design may be employed ►A research question should address only 1 concept ►Question must be measurable

Types of Questions Asked ►Once you have identified the topic of study, you will need to consider the type of question you want answered and how it will be answered ►Two paradigms ● Quantitative Paradigm § Generally attempt to quantify variables of interest. Questions frequently address “how well or how much. ”

Types of Questions Asked ● Qualitative Paradigm § “there are times when we wish to know not how many or how well, but simply how. ” (Shulman, 1988, pg. 7)

Class Exercise ►Now you’re ready to formulate your own research question(s) ►Sample questions: ● Is there a relationship between participation in an Elluminate chat session and course grade? ● How do 5 th grade students experience the anticipation of standardized testing?

Research Questions ► From Topic to Research Question A good research topic asks a clear, concise question. Asking a research question helps you keep a tight focus on your topic. ► Tweaking Your Research Question A good research topic is broad enough to allow you to find plenty of material, but narrow enough to fit within the size and time constraints of your paper. ● If your topic is either too broad or too narrow, consider adding or eliminating the following elements: Time Period, century, decade, future, Population Type, age, gender, nationality, species, Geographic Location country, state, region, Point of View economic, social, cultural, biological

Assignment 2 Components (see syllabus for details) ►Title Page ►Nature of the Problem ►Background and Significance of the Problem ►Literature Review ►Research Questions ►References

Topic ideas ►Online chat reference ● Types of questions § Subject? Type? § # of turnaways* ● Difference in discourse § In-person vs. chat ● Partnership studies § Similar libraries with same software

Topic Ideas ►E-book usage ►Usability studies of ● Online tutorial(s) ● ‘My Library” portals ►Analysis of library web sites or library instruction sites or pathfinders by best practices ►Student learning outcomes in LI programs

Types of methodologies ►Qua. Litative Measures ● Descriptive ● Numbers not the primary focus ● Interpretive, ethnographic, naturalistic ►Qua. Ntitative Measures ● N for numbers ● Statistical ● Quantifiable

Qua. Litative measures ►Content Analysis ● Analyzed course syllabi of library use through discipline and level (Rambler) ● Studied online tutorials, applying best practices recommendations (Tancheva)

Qua. Litative Measures ►Discourse Analysis ● Analyzed student responses in writing and discussions to a short film & compared findings to parallel study with LIS grad Ss (Vandergrift) ►Focus Groups ● Discussed how participants experience & use the library (Von Seggern & Young) ● Studied why students use the Internet and how much time they use it (Wilson)

Qua. Litative Measures ►Interviews ● Studied 25 HS students’ web use for research assignments (Lorenzen) ● Looked at what type of information first year students need and how they go about acquiring it (Seamans) ►Observation (obtrusive) ● Observed students as they conducted online research & noted their activities (Dunn) ►Observation (Unobtrusive) ● Retrieval of discarded cheat sheets to analyze academic misconduct (Pullen et. al. )

Qua. Litative Measures ►Think Aloud Protocols ● Studied how users navigate a library web site (Cockrell & Jayne) ►Usability testing ● Examined students’ mental models of online tutorials (Veldof & Beavers)

Qua. Ntitative measures ►Compare. Things ►Count Things ►Survey People About Things p <. 05

Qua. Ntitative measures p <. 05 ►Comparison studies ● Experimental and control groups ● Instructional methodologies (Colaric; Cudiner & Harmon) ● Program assessment using before/after analysis of research papers(Emmons & Martin)

Qua. Ntitative measures p <. 05 ►Pre & Post Tests (Van Scoyoc) ►Measures & Scales § Bostick’s Library Anxiety Scale (Onwuegbuzie & Jiao; Van Scoyoc) § Procrastination Assessment Scale (Onwuegbuzie & Jiao)

Qua. Ntitative measures p <. 05 ►Numeric Studies ● Citation Analysis Bibliometrics (Dellavalle) ● Webometrics (Bar-Ilian)

Ready Made Data Sets ►National Survey of Student Engagement (Whitmire) ►College Student Experiences Questionnaire (Kuh and Gonyea) ►The Web ● Internet Archive (Ryan, Field & Olfman) ● Electronic journals (Dellavalle) ►Library server logs

Common Pitfalls ►Problems with population ● Sampling? § Representativeness? § Self-selection?

Research Problem #1 A study assessing student learning outcomes in 2 broad categories (concepts, techniques) by examining student research journals in 1 section of an elective information literacy course in fall semester.

Research Problem #4 A 2004 article on a library use and services satisfaction study that used as its measurement tool a survey given to every nth person entering the library building on 40 randomly selected days throughout the school year.

Research Problem #5 An outcomes assessment research project of a 5 year old IL program in which all incoming freshmen must participate. Total student population on campus is divided between 32% freshmen to senior (or 4 year) and 68% transfer students.

Common Pitfalls ►Problems with operationalization ● Defining of what is measured

Research Problem #2 An experimental study that proposes a fund allocation formula for academic library collections based on the following: average of overall book price + average of overall serial prices * degree level (10 for undergraduate to 30 for doctorate) / the number of students enrolled in degree program as majors + the total number of faculty in the department * three * total number of students in program. (OAB + OAS) * D/(Sn +(Fn*3))*Sn N. B. Not a standard formula

Research Problem #3 A newspaper article you read just the other day stated that in a recently published study done at a major U. S. university, researchers found that domestic violence affects 1 in every 4 women.

Research Problem #4 A 2004 article on a library use and services satisfaction study that used as its measurement tool a survey given to every nth person entering the library building on 40 randomly selected days throughout the school year.

Research Problem #5 Over a one year period, researchers studied the occurrence of turn-aways in a virtual reference service and noted that the significantly high occurrence of turn-aways indicates increased need for virtual reference service.

Common Pitfalls ►Problems with generalizability ● False conclusions ● Transformations

Research Problem #1 A study assessing student learning outcomes in 2 broad categories (concepts, techniques) by examining student research journals in 1 section of an elective information literacy course in fall semester.

Research Problem #7 A survey of faculty found that the majority of those interviewed interacted most with librarians at the reference desk. The researchers concluded that most faculty view librarians in a servile role.

Keep In Mind That ►No study is perfect ►“All data is dirty is some way or another; research is what you do with that dirty data” (Manuel) ►Measurement involves making choices

Be Critical About Numbers (Best 2001) ►“Every statistic is a way of summarizing complex information into relatively simple numbers. ” (Best) ►How did the researchers arrive at these numbers? ►Who produced the numbers and what is their bias? ►How can key terms be defined & in how many different ways?

Be Critical About Numbers ►How was the choice for the measurement made? ►What type of sample was gathered & how does that affect result? ►Is the statistical result interpreted correctly? ►If comparisons are made, are they appropriate? ►Are there competing statistics?

Getting Started ► Read to learn; read to analyze ● About research methodology ● Studies on similar topics ● Interesting studies ● Non-library studies

Getting Started ►Finding a topic needn’t be traumatic ● Work projects Research studies § P&T overhaul § Library GO Bond Proposal Project § Library workshop trends § User repair strategies

Getting Started ►Data collection involves agreement & consent ►Forge partnerships ►At some point you will need to leave the comfort zone of reading and literature gathering and …

Just get out and do it!

Questions?

Research methodology Quantitative Methods Qualitative procedures

Quantitative Methods ►A definition ● A survey or experiment that provides as output a quantitative or numeric description of some fraction of the population, called the sample.

Components of a survey method ►The survey design ►The population and sample ►The instrumentation ►Variables in the study ►Data analysis

The survey design ►Purpose of the survey ►The research question ►Type of survey ● Cross sectional ● Longitudinal ►Form of data collection

The population and sample ►Description of the population ►Sampling design ● Single stage ● Multistage ● Stratified ►Sample selection

The instrumentation ►The instrument (tool) ● Existing ● New ►Rating scale ● Likert scale: Rating the Items. 1 -to-5 rating scale where: 1. 2. 3. 4. 5. = strongly unfavorable to the concept = somewhat unfavorable to the concept = undecided = somewhat favorable to the concept = strongly favorable to the concept ►Pilot ►Administration ● Postal survey ● email

Variables and analysis ►The research question ►Variable in the research ● E. g. Number of years of academic study ►The questions in the instrument ● E. g. How many years of study in a University § As an undergraduate? § As a postgraduate? ►Data analysis ● Steps ● Bias in the data § Non-response ● Statistics, e. g. mean, standard deviation etc.

Components of an experimental method ►Subjects ►Instruments and materials ►The experimental design

Subjects ►Selection ● Conveniently ● Random (RCT) ►Group assignment ● Random ● Matched. E. g. Ability, Age ● Size ►Variables ● Dependent ● Independent

Randomized Controlled Trial (RCT) A true experiment, in which the researcher randomly assigns some patients to at least one maneuver (treatment) and other patients to a placebo, or usual treatment. Key features = the classic way to evaluate effectiveness of drugs (or exercise, diet, counseling). Patients are followed over time (Prospective). If properly done, an RCT can be used to determine cause and effect

Instrumentation and Materials ►Description ►Validation ● Pilot ● Content validity ● Prediction validity ►Materials

The experimental design ►Type ● Pre-experimental § No control group ● Quasi-experimental § Control group, but not randomly assigned ● Single subject design (over time) ● Pure experiment ● Repeated measures § Change groups

Overview of Qualitative Research Design • Historical routes in anthropology • Generates new understanding by naming and framing concepts and themes • Removes bias by questioning preconceived assumptions of the social group under study • Promotes neutrality through adoption by the researcher of naïve stance or critical discussion, challenges pre-conceived assumptions of both the researcher and the social group under study • Produces new understanding about the world, changes the way power, culture and social interaction are understood

Data Collection in Qualitative Research • Observation (Videoed, non-participant, semiparticipant and participant observation, field notes) • Interviews (individual and group - known as focus groups, tape recorded and transcribed, field notes) • Secondary data analysis (using written material collected for purposes other than research) • Questionnaires (unstructured, postal, interviews) • A mixture of all four

Questions in Qualitative Research In qualitative research questions are open-ended. Sometimes a check list or topic guide will be used by the researcher to ensure all the relevant areas are covered. This is known as semi-structured data collection. It is used in all four methods of data collection Sometimes the only guide is the topic itself and the researcher collects verbatim or naturally occurring data. This is known as unstructured data collection. It is used in all four methods of data collection

Sampling in Qualitative Research The sampling method of choice is theoretical sampling (queuing behaviour) However, often this is not possible and people resort to convenience sampling (students) and snowball sampling (mental health in black and ethnic minority communities) Neither of the latter two methods are considered strong but maybe all that can be achieved. Research must be viable.

Data Analysis in Qualitative Research • Read and re-read data, become engrossed in it. • Identify themes: common, conflicting, minority • Test themes across the data set, where are they common, under what circumstances are they found, not found. This sets the parameters on the interpretation and generalisation of data • Get more than one person to analyse the data independently then together • Demonstrate trustworthiness in data analysis • Examples • Biographical continuity • Nursing routines as a method of managing a transient workforce

Qualitative research ►Interpretative research ►Process orientated ►Researcher(s) are the primary data collection instrument ►Descriptive research ►Outputs are an inductive process

References ►MSc project web pages http: //www. comp. glam. ac. uk/gis/start. asp? whatfile=gis/gis rc/msc-proj. htm ►Creswell, J. W. (1994) Research design : qualitative and quantitative approaches. - Thousand Oaks, Calif. ; London : Sage Publications, ISBN 0803952546