NonExperimental Quantitative Research Designs Chapter 8 This multimedia

Discussion Topics l Non-experimental research designs – – – l Descriptive designs Relationship designs

Non-Experimental Designs l l Research design - the plan and structure of research to

Non-Experimental Designs l Four types of designs – – Descriptive Relationships l l –

Descriptive Designs l Studies that describe a phenomena – Statistical nature of the description

Descriptive Designs l Criteria for evaluating descriptive studies – – Conclusions about relationships should

Relationship Designs l Two types of designs – – Comparative Correlational Copyright © Allyn

Comparative Designs l These designs investigate the relationship of one variable to another by

Comparative Designs l Criteria for evaluating these designs – – Participants and instruments are

Correlational Designs l Simple correlation designs – – Designs that examine the relationship between

Correlational Designs l Simple correlation designs – Cautions in interpreting correlations l l A

Correlational Designs l Simple correlation designs – Cautions in interpreting correlations l l Lack

Correlational Designs l Prediction designs – – Designs that examine the predictive nature of

Correlational Designs l Prediction designs – Simple predictive studies l l Performance on one

Correlational Designs l Prediction designs – Simple predictive studies l Factors influencing correlations Possibility

Correlational Designs l Multiple regression – – Studies that examine performance on several variables

Correlational Designs l Logistic Regression – – Another type of multiple regression analysis used

Correlational Designs l Multiple regression – Issues of concern l l Sample size of

Correlational Designs l Significance of correlation coefficients – Statistical significance l l – Does

Correlational Designs l Guidelines for interpreting the size of correlation coefficients – Much larger

Correlational Designs l Guidelines for interpreting the size of correlation coefficients – Multiple correlation

Correlational Designs l Criteria for evaluating correlational studies – – – Causation should not

Correlational Designs l Criteria for evaluating correlational studies – – – The size of

Causal Comparative Designs l Ex-post-facto designs – Studies that investigate the relationships between independent

Causal Comparative Designs l Ex-post-facto designs – Issues of concern l l Selecting participants

Causal Comparative Designs l Correlational causal-comparative studies – – Use of correlational models to

Causal Comparative Designs l Criteria for evaluating causal-comparative studies – – – Primary purpose

Using Surveys l A data collection method that is very useful in descriptive and

Using Surveys l Cross sectional designs – – Information is collected from one or

Using Surveys l Longitudinal designs - information is collected from the same participants over

Using Surveys l Steps in designing a survey – – Define a purpose and

Using Surveys l Steps in designing a survey – Develop the items – guidelines

Using Surveys l Steps in designing a survey – Develop directions l l –

Using Surveys l Steps in designing a survey – Pilot test l l 15

Using Surveys l Response rates – – Low response rates are the major limitation

Using Surveys l Using electronic surveys – Internet based surveys l l – –

Using Surveys l Using electronic surveys – Advantages l l – Reduced time and

Slides: 37

Download presentation

Non-Experimental Quantitative Research Designs Chapter 8 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: any public performance or display, including transmission of any image over a network; preparation of any derivative work, including the extraction, in whole or part, of any images; any rental, lease, or lending of the program. Copyright © Allyn & Bacon 2008

Discussion Topics l Non-experimental research designs – – – l Descriptive designs Relationship designs Causal-comparative and Ex Post Facto designs Using surveys in quantitative research Copyright © Allyn & Bacon 2008

Non-Experimental Designs l l Research design - the plan and structure of research to provide a credible answer to a research question Purpose of non-experimental designs – Describe current existing characteristics such as achievement, attitudes, relationships, etc. Copyright © Allyn & Bacon 2008

Descriptive Designs l Studies that describe a phenomena – Statistical nature of the description l l l Frequency Percentages Averages Graphs Importance of these designs in the early stages of the investigation of an area Copyright © Allyn & Bacon 2008

Descriptive Designs l Criteria for evaluating descriptive studies – – Conclusions about relationships should not be drawn Participants and instruments should be described completely Copyright © Allyn & Bacon 2008

Comparative Designs l These designs investigate the relationship of one variable to another by examining differences on the dependent variable between two groups of participants – – If math scores for males are significantly higher than those for females, a relationship exists between gender and math achievement If the academic self-concept scores for ninth graders are significantly different than those for twelfth graders, a relationship exists between grade level and academic selfconcept Copyright © Allyn & Bacon 2008

Comparative Designs l Criteria for evaluating these designs – – Participants and instruments are described completely Criteria for identifying the different groups is clearly stated No inferences are made about causation Graphs and images depict the results accurately Copyright © Allyn & Bacon 2008

Correlational Designs l Simple correlation designs – – Designs that examine the relationship between two variables Two variables l l – Predictor and criterion Use caution describing the variables as independent and dependent Examples l l Math achievement and math attitudes Teacher effectiveness and teacher efficacy Copyright © Allyn & Bacon 2008

Correlational Designs l Simple correlation designs – Cautions in interpreting correlations l l A relationship between two variables (e. g. , achievement and attitude) does not mean one causes the other (i. e. , positive attitudes do not cause high levels of achievement) Possibility of low reliability of the instruments makes it difficult to identify relationships Copyright © Allyn & Bacon 2008

Correlational Designs l Simple correlation designs – Cautions in interpreting correlations l l Lack of variability in scores (e. g. , everyone scoring very, very low; everyone scoring very, very high; etc. ) makes it difficult to identify relationships Large sample sizes and/or using many variables can identify significant relationships for statistical reasons and not because the relationships really exist Copyright © Allyn & Bacon 2008

Correlational Designs l Prediction designs – – Designs that examine the predictive nature of the relationships between variables Two types of designs l l Simple prediction Multiple regression Copyright © Allyn & Bacon 2008

Correlational Designs l Prediction designs – Simple predictive studies l l Performance on one variable (i. e. , the predictor) is used to predict performance on a second variable (i. e. , the outcome or criterion) Examples Scholastic Aptitude Test (SAT) scores are used to predict college freshmen grade point averages – Scores from a mathematical attitude scale are used to predict math achievement scores – l Importance of the time interval between collecting the predictor and criterion variable data Copyright © Allyn & Bacon 2008

Correlational Designs l Prediction designs – Simple predictive studies l Factors influencing correlations Possibility of low reliability of the instruments measuring the predictor and criterion variables makes it difficult to identify relationships – Length of time between the predictor and criterion variable data collection – Existence of many factors, not only the one being examined, that influence the criterion variable – Copyright © Allyn & Bacon 2008

Correlational Designs l Multiple regression – – Studies that examine performance on several variables (i. e. , predictor variables) to predict performance on a single outcome variable (i. e. , criterion) Examples l l Scholastic Aptitude Test (SAT) scores, high school grade point average, and high school rank in class are used to predict college freshmen grade point average Math attitude scale scores, academic self-esteem scale scores, and prior math grades are used to predict math achievement scores Copyright © Allyn & Bacon 2008

Correlational Designs l Logistic Regression – – Another type of multiple regression analysis used to examine the relationship between the predictor variables and dependent variable. The result is a prediction of whether the participant is a “case” or “non-case” l Example: Overall GPA, gender, and special education classification used to determine if a student will pass or fail a standardized test. Copyright © Allyn & Bacon 2008

Correlational Designs l Multiple regression – Issues of concern l l Sample size of at least 10 subjects for each predictor variable Relationships among the predictor variables (i. e. , colinearity) Copyright © Allyn & Bacon 2008

Correlational Designs l Significance of correlation coefficients – Statistical significance l l – Does a statistical relationship exist? Is the observed correlation significantly different from zero? Practical significance l l Does a relationship of practical importance exist? Coefficient of determination (r 2) - the percentage of the criterion variable variation that can be explained by the variation in the predictor variable Copyright © Allyn & Bacon 2008

Correlational Designs l Guidelines for interpreting the size of correlation coefficients – Much larger correlations are needed for predictions with individual than with groups l l – Crude group predictions can be made with correlations as low as. 40 to. 60 Predictions for individuals require correlations above. 75 Exploratory studies l l Correlations of. 25 to. 40 indicate the need for further research Much higher correlations are needed to confirm or test hypotheses Copyright © Allyn & Bacon 2008

Correlational Designs l Guidelines for interpreting the size of correlation coefficients – Multiple correlation coefficients of. 20 -. 40 are common and usually indicate practical significance Copyright © Allyn & Bacon 2008

Correlational Designs l Criteria for evaluating correlational studies – – – Causation should not be inferred from correlational studies The reported correlation should not be higher or lower than the actual correlation Practical significance should not be confused with statistical significance Copyright © Allyn & Bacon 2008

Correlational Designs l Criteria for evaluating correlational studies – – – The size of the correlation should be sufficient for the use of the results Prediction studies should report the accuracy of predictions for new subjects Procedures for collecting data should be clearly indicated Copyright © Allyn & Bacon 2008

Causal Comparative Designs l Ex-post-facto designs – Studies that investigate the relationships between independent and dependent variables in situations where it is impossible or unethical to manipulate the independent variable l Example - what is the effect of pre-kindergarten (Pre-K) attendance on first grade achievement Cannot mandate Pre-K attendance for children – Characteristics and resources of families who do and do not send their children to Pre-K may influence first grade achievement – – Similarities with correlational and experimental research designs Copyright © Allyn & Bacon 2008

Causal Comparative Designs l Ex-post-facto designs – Issues of concern l l Selecting participants who are as similar as possible on all characteristics except the independent variable Generalizing beyond the participants studied Copyright © Allyn & Bacon 2008

Causal Comparative Designs l Correlational causal-comparative studies – – Use of correlational models to investigate possible cause and effect relationships Sophisticated statistical models l l – Path analysis Structural equation modeling Fundamental limitations of all correlational research designs apply Copyright © Allyn & Bacon 2008

Causal Comparative Designs l Criteria for evaluating causal-comparative studies – – – Primary purpose is to investigate causal relationships when experimental designs are not possible Presumed causal condition has already occurred Potential extraneous variables are considered Existing differences between groups being compared are controlled Causal conclusions are made with caution Copyright © Allyn & Bacon 2008

Using Surveys l A data collection method that is very useful in descriptive and correlational studies – – – l Versatile Efficient Generalizable Two types of survey designs – – Cross sectional designs Longitudinal designs Copyright © Allyn & Bacon 2008

Using Surveys l Cross sectional designs – – Information is collected from one or more groups at the same time Examples l l – Student’s, teacher’s, administrator’s, and parent’s opinions regarding an extended school year Elementary, middle, and secondary teachers’ feelings toward a new school board policy Issue of concern - comparisons across groups can be the result of differences between participants within the groups l Fifth and seventh graders opinions can be affected by a changes in the attendance zones of a school Copyright © Allyn & Bacon 2008

Using Surveys l Longitudinal designs - information is collected from the same participants over time – – Example - changes in the academic self-concept of students from the sixth to the twelfth grade Issues of concern l l Loss of subjects over time Difficulty tracking participants over time Copyright © Allyn & Bacon 2008

Using Surveys l Steps in designing a survey – – Define a purpose and objectives Identify the resources needed and the target population l l l – Cost of preparation, printing, mailing, and analyzing results Time needed to administer the survey Sample size Choose the method l l Paper Electronic Telephone Interview Copyright © Allyn & Bacon 2008

Using Surveys l Steps in designing a survey – Develop the items – guidelines l l l – Use clear, unbiased, non-ambiguous language Keep it short and simple Use grammatically correct language Do not write leading items Use the same response scale for all items Be consistent with wording Design the format of the survey l l White space Font size Copyright © Allyn & Bacon 2008

Using Surveys l Steps in designing a survey – Develop directions l l – Make it clear with no ambiguity Indicate clearly how participants are to respond Indicate where responses are recorded Indicate what participants should do when finished Develop a letter of transmittal l Keep it brief Include a statement of the purpose of the research Include a statement of the benefits of the research Copyright © Allyn & Bacon 2008

Using Surveys l Steps in designing a survey – Pilot test l l 15 -20 representative participants Identify concerns – – – Clarity Format Responding Directions Time to complete Copyright © Allyn & Bacon 2008

Using Surveys l Response rates – – Low response rates are the major limitation of survey use Suggestions for increasing response rates l l l Design the survey well Contact the subjects several times especially following-up on nonrespondents Include a self-addressed return envelope Use a good transmittal letter Use a telephone for follow-up Use incentives for completing the survey Copyright © Allyn & Bacon 2008

Using Surveys l Using electronic surveys – Internet based surveys l l – – E-mail attachments Web pages Evaluations of online university survey research centers Criteria for evaluating survey research centers Copyright © Allyn & Bacon 2008

Using Surveys l Using electronic surveys – Advantages l l – Reduced time and cost Easy access Quick responses Ease of creating data sets Disadvantages l l Limited to those with access to the technology Confidentiality and privacy issue Copyright © Allyn & Bacon 2008