DEVELOPING RUBRICS Dr Jesus R Dela Rosa Instructor

DEVELOPING RUBRICS Dr. Jesus R. Dela Rosa Instructor, English Language Center JIC ABET Member December 05, 2011

Outline �Objectives �Introduction �Discussion Rubrics Proposed rubrics for � Oral presentation; Lab work & Writing report �Academic Exchanges

Objectives Discuss rubrics briefly � Explain a proposed rubric for oral presentation � Gather feedback on its content and form from the participants �

Introduction �Delivery of AS programs � Different courses= different assessment tools � Bloom’s learning domains Cognitive (head), affective(heart), psychomotor (hands)

Introduction � Different tools to assess outcomes � � � Direct = What students/learners can do Indirect= What other people think and feel about students/learners can do/have (? ) Criterion 4. Continuous Improvement � Upper State U model of the Self-Study Report (SSR) 41 pages out of 100 (41%) how we measure student outcomes/performance through assessment tools clearly specified in our Course Assessment Charts Some reminders to SSR writers Use of various assessment tools (direct/indirect ) Use of rubrics is encouraged in some courses

Discussion � Why rubrics �A hype? (hot issue) �From test to task (performance-based not knowledge-based) “Tell me-I listen; Teach me- I learn- Show me how- I learn and live” �ABET – outcome-based Does not believe in “one size fits all” assessment (triangulation)

Why rubrics �Written tests- a thing of the past? �Alternative assessment tools � The bottom line �Rubrics-learning grounded in the real world where students live Student-centered; standards driven (criterion-referenced)

What are Rubrics � Rubrics - performance-based assessments that evaluate student performance on any given task/set of tasks that ultimately leads to a final product, or learning outcome (over-all assessment)= Student Outcomes use specific criteria as a basis for evaluating or assessing student performances (specific) marking sheet = subjective –more objective �narrative descriptions (descriptors) of possible performance related to a given task � http: //www. teach-nology. com/currenttrends/alternative_assessment

Three components of a rubric

Rubrics � Types of rubrics (according to rating scales) �Holistic- More global Views the final product as a set of interrelated tasks contributing to the whole Anchor points are used to assign value to descriptions of products or performances that contribute to the whole

Holistic rubric Holistic scoring proves to be efficient and quick One score provides an overall impression of ability on any given product or work Often written generically and can be used with many tasks Saves time by minimizing the number of decisions raters must make Raters if trained properly tend to apply them consistently, resulting in more reliable measurement �

Holistic rubric �Disadvantages Scoring does not provide detailed information about student performance in specific areas of content or skill No specific feedback about students’ strengths/weaknesses Does little to separate the tasks Performances may meet criteria in two or more categories, making it difficult to select the one best description Criteria cannot be differentially weighted

Holistic rubric: A sample Oral Presentation Scoring Rubric Mastery =usually makes eye contact; volume is always appropriate ; enthusiasm present throughout presentation; summary is completely accurate Proficiency=usually makes eye contact; volume is always appropriate; enthusiasm is present in most of presentation; only one or two errors in summary Developing=sometimes makes eye contact; volume is sometimes appropriate; occasional enthusiasm in presentation; some errors in summary Inadequate=never or rarely makes eyes contact; volume is inappropriate; rarely shows enthusiasm in presentation; many errors in summary

Analytic rubric � Analytic- � Scoring breaks down the objective or final product into component parts. � Each part is scored independently. � The total score is the sum of the rating for all of the parts that are being evaluated. � Useful in giving feedback on areas of student performance (strengths/weakness). � Dimensions can be measured to reflect relative importance. � Progress over time can be demonstrated when used repeatedly.

Analytic rubric � Disadvantages �More time to prepare (ask any JIC ABET member!) �More possibilities for raters to disagree �More difficult to achieve intra- and inter-rater reliability on all of the criteria/dimensions

Analytic rubric: A sample Criteria Never Sometimes always Score Makes eye contact 0 3 4 Volume is appropriate 0 2 4 Enthusiasm is evident 0 2 4 Summary is accurate 0 4 8 Total

Holistic or analytic? � � � Left to the better judgment of the experts 6 criteria at most (holistic); more criteria (analytic) Whether holistic or analytical scales � Important factors in developing effective rubrics- Use of clear criteria that will be used to rate a student's work. The performance being evaluated is directly observable. Students should be informed as to what criteria they are being held accountable.

Proposed rubric for oral presentation. Rbrc jessedit. docx � � � Analytic as to type (7 criteria-more ? less? ) Criteria mapped to performance indicators (PIs) -JIC ABET (establish consistency) Four scales (1 -Beginning; 2 -Developing; 3 Competent; 4 -Outstanding) Provision for scoring each criterion Assessment report for oral presentation Results form part of applicable PIs & SOs

Validation and Reliability Concerns � � Rubrics just like other assessment tools have validation and reliability concerns AS programs may be left to decide or JICABET � Content validity Let colleague(s) review your rubric (documented/recorded) Inform students how it works (beginning of semester? ) Check if it is manageable (pilot-testing; to be documented)

Summary � � � AS programs have different courses Different courses have different learning domains Rubrics as direct assessment tools � why, what, types, differences, advantages/disadvantages and which rubric to use � � Proposed rubric for oral presentation prepared by JIC ABET- parts; assessment report Validity and reliability concerns in future � Content validity (colleagues & students) � Other validation/reliability concerns

Conclusions � � Rubrics are direct assessment tools that are necessary to measure learning outcomes. Rubrics require some time to develop and validate. There are validity and reliability concerns on the development and administration of rubrics. JIC-ABET Committee has prepared rubrics for guidance of/fine-tuning by the AS programs.

Proposed Rubric for Lab Works SN Criteria Performance Indicators Measuring Methods 1 Beginning 2 Developing 3 Competent Experiment is generally carried out with attention to relevant safety procedures. - 1 2 Safety Rules Identification of Equipment c. 1 c. 2 Observations Exams 3 Use of Equipment c. 2 Observations Exams - 4 Experimental Procedures c. 3 i. 2 i. 3 - Safety procedures were ignored. Always needs assistance. - Fails to identify suitable equipment most of the time. Always needs assistance. Fails to use equipment properly. Always needs assistance. Observations Exams Fails to read Lab manuals and follow experimental procedures. Always needs assistance. - Experiment is carried out with some attention to relevant safety procedures. 4 Outstanding Score - Experiment is carried out with full attention to relevant safety procedures. Identifies suitable equipment sometimes. Some assistance is needed. - Uses equipment properly most of time. Some assistance is needed. Reads Lab manuals and follows experimental procedures most of the time. Some assistance is needed. - Always Identifies suitable equipment. - - Always uses equipment properly. Always read Lab manuals and follows experimental procedures. Always Identifies equipment and explores other equipment. Always uses equipment properly and explores it. Always read Lab manuals in advance, follows experimental procedures and suggests improvements.

SN Criteria Performanc e Indicators Discipline d. 3 h. 3 Measuring Methods 1 Beginning 5 Observations - 6 Punctuality i. 2 Comes late or does not attend the lab session most of the time. Always misses the deadlines. Observations SIS attendance records - 7 Participation d. 1 d. 2 Observations - Does not abide by the rules most of the time. Does not perform any duties of assigned team role. Always relies on others to do the work. 2 Developing - Sometimes abides by the rules. 3 Competent Abides by the rules most of the time. - Comes late sometimes. Misses the deadline sometimes. - Performs very little duties. Rarely does the assigned work-often needs reminding. - Always comes on time. Always meets the deadlines. Performs nearly all duties. Usually does the assigned work--rarely needs reminding. 4 Outstanding - Scor e Always abides by the rules most of the time - Always comes on time. Always ahead of the deadline. - - - Performs all duties of assigned team role. Always does the assigned work without having to be reminded.

Proposed Rubric for Project Reports SN 1 Criteria Introduction Performance Indicators e. 1/e. 2 Measuring Methods Review reports 1 Beginning - 2 Developing Introduction is missing. Introduction is confusing. 3 Competent - 4 Outstanding Score Introduction is adequate. - Introduction provides background a forecast of the document. Problem or situation is defined clearly with orienting material for audience 2 Organization I. 1/I. 2/a. 1/a. 2 Review reports - Points are not ordered. Report is disorganized and layout is somewhat weak. Most points are ordered well. No major problems with layout. - Points are clearly presented in a logical order. Easily followed. Page layout is effective. 3 Language f. 3 Review reports - Choice of words is poor. Grammar, spelling and punctuation are poor. Choice of words is sometimes not appropriate. Grammar, spelling and punctuation are not satisfactory. Choice of words is appropriate most of the time. Minor problems with grammar, spelling, punctuation. - Wording is concise, clear, and easy to follow. Consistently proper grammar, spelling and punctuation. Most elements are missing or erroneous/not related. Some elements are missing or inadequately discussed. Most of the elements are adequately discussed. - All of the elements are adequately discussed and supported with additional relevant information Clear, insightful conclusions. - 4 Content I. 1/i. 2/i. 3/g. 1 g. 2/b. 1/b. 2/b. 3 a. 1/a. 2 Review reports - - - 5 Conclusions e. 1/e. 2 Review reports - Conclusions are missing. - Conclusions are inadequate. - Most but not all points contained in the conclusion 6 Visuals f. 2/g. 2/a. 1/ a. 2 Review reports - Graphs/tables /figures/pictur es are missing. Graphs/tables/ figures/pictures are inadequate. - Graphs/tables/ figures/pictures are adequate, well labeled and presented. 7 References h. 1/f. 4/g. 1/ g. 2/a. 1/a. 2 Review reports - References are notlisted. References are used and referred and listed properly. - References are extensively used and properly listed. - Some references are used but not referred to or listed properly.

THANK YOU!