The Roles of Rater Goals and Ratee Performance

The Roles of Rater Goals and Ratee Performance Levels in the Distortion of Performance Ratings BY XIAOYE MAY WANG KIN FAI ELLICK WONG, AND JESSICA Y. Y. KWONG 2010 CHELSEA HUTTO

Overview � 2 studies �Peer rating and non-peer rating context �Results found: raters use different rating tactics to achieve specific goals

Rating Inaccuracy �One of the most important concerns in performance evaluation �Traditionally thought to be a result of rating errors Assumes raters involuntarily make errors when rating, due to poor scale design or cognitive limitations

A More Current Viewpoint �Considering whether raters are intentionally distorting ratings �Suggest raters have goals in mind and provide ratings that are consistent which such goals AKA Goal based approach to performance evaluations

Current Viewpoint �Not just a measurement process �And not just a bunch of lazy raters! �Must look at rater goals & rater characteristics �Ratings may vary based on interaction of rater goals and rater performance

This particular study �Wong and Kwong (2007) �Examine the interaction between rater goals and performance levels on performance ratings �Why examine performance level? ? Never been formally tested & functionality of performance evaluation

Current Research �Purpose: extend previous research in a context that allows us to understand the effects of rater goals on rating scores for ratees with different levels of performance. � Examined the four most common rater goals

Rater Goals �Identification: ID strengths and weaknesses �Harmony: Seeks to maintain group harmony and interpersonal relationships �Fairness: Seeks to reflect the accurate contribution of each team member �Motivating: Seeks to increase the future motivation of ratees

Goal and Performance Interaction �Harmony: Fewer rewards = lower satisfaction and more antagonistic behaviors �Low performers vs. high performers �Hypothesis 1: Raters will exhibit rating inflation when they pursue a harmony goal (vs. ID goal); with the inflation being more pronounced as the performance level of the ratee decreases

Goal and Performance Interaction �Fairness: Based of equality and equity norms �Equity vs. equality orientation � Hypothesis 2: In the context of pursuing fairness (vs. identification), raters will inflate ratings for poorer performers and deflate ratings for better performers when they evaluate performers from the same working group; however, raters will have no such rating distortion when they evaluate performers as out-group members.

Goal and Performance Interaction �Motivating: Raters are likely to have different theories of motivation for low and high performers, thus raters believe that different types of performance feedback are required to motivate. �Low vs. High performers �H 3: Raters will inflate ratings of poorer performers and deflate ratings of better performers when they have a motivating goal (vs. an identification goal).

Study One � 103 Undergrad students in a HRM course �Within participants design �Groups were formed voluntarily �Required to complete peer evaluations twice Mid-semester and End of semester

Materials � 5 page questionnaire � 4 sets of peer evaluations, asked to complete with a different goal in mind �Rated on a 1 -7 contribution scale

Analyses � 2 separate analyses �Hierarchical linear modeling (HLM) �ANOVA 2 x 3 study design Goal Condition x Performance Level

Results for Harmony Goal �Hypothesis 1: Significance found at mid-semester and end of semester between harmony goal and rating scores �Interaction between harmony and ratees performance level was also significant �Greater rating inflation as performance level decreased Inflation was greatest for low performers, followed by medium, and then high

Results for Fairness Goal �Hypothesis 2: Significance was found between fairness and rating scores for mid-semester and end of semester �Interaction between fairness and performance level was significant – indicates inflation varied across performance levels �Effects of fairness was significant for low and medium, but not for high performers

Results for Motivating �Hypothesis 3: Significance found between motivating and rating scores at mid-semester and end of semester �Interaction between motivating and performance level was significant �Effect of motivating was significant for low and medium, but not for high performers

Discussion �Raters have different distortions for ratees with various performance levels under different goal conditions �Study 1 supports general proposition that to achieve a specific goal, raters tailor the degree of rating distortion to ratees with different performance levels �Raters tended to inflate more as performance levels decreased

Study 2 � 120 Undergrad students in an organizational behavior course �Between subjects design �Randomly assigned to one of the rater goal conditions � 15 minute video and evaluated the performance of 6 team members on a 10 item scale

Analyses and Results �Same as Study 1 �Hypothesis 1: Not supported �Interaction between rater goals and ratee performance was not significant under harmony goal condition Supported by both ANOVA and HLM

Results �Hypothesis 2: Partially supported �Interaction between fairness and ratee performance level was significant �Effect was significant for high performers but not for medium or low Significant rating deflation for high performers

Results �Hypothesis 3: Partially supported �Interaction between motivating and performance level was significant �Effect of motivating was significant for low but there was not significant distortion for medium or high performers

Differences between Studies �Harmony condition: Found significant results in Study 1 but not Study 2 �Inconsistency may be due to different relationships between raters and ratees. �Goal may not have been as salient in Study 2 as in Study 1

Discussion for both studies �Harmony: Raters deflated their ratings for all performers. Raters tried to reduce conflict by lowering the discriminability rather than inflating ratings �Fairness: Raters who wanted to achieve a motivating goal inflated ratings for low but didn’t distort for high Results suggest people are more likely to adopt equality rule for low performers in peer rating context and high performers in the nonpeer rating context

Discussion �Results suggest rating accuracy is shaped by the ratees ability (performance level) and the raters motivation (goals) �Rating errors are not only due to inadequate rating skill, but also a result of the intentions of the raters �Demonstrated the importance of examining contextual effects on performance appraisal �Suggests there is an alternative understanding of rating inaccuracy

Limitations �Didn’t build on long term relationships (one semester and 45 minutes) �Lack of politics that are present in an organization �Study 1 was a single item measure – hindered exploration of performance dimensionality

Future Research �Should explore individual differences in implicit theories of rating strategies �Clearly stating the rating goal for the rater may be a solution to reduce rating discrepancies

Discussion Questions �The authors primarily used their own research to base research design and decisions on, do you feel this could significantly affect the overall results of this study? �In what ways could we measure rater motivation to reduce rating inaccuracy in the organization? �Do you think the authors research design could be improved to more appropriately test the author’s hypotheses? If so, how?