Performance Measurement in Government Programs Current Status and

  • Slides: 18
Download presentation
Performance Measurement in Government Programs: Current Status and Future Applications Dr. Julia H. Bryan

Performance Measurement in Government Programs: Current Status and Future Applications Dr. Julia H. Bryan College of Public Affairs Doctor in Public Administration University of Baltimore (2013 Graduate) October 19, 2013

Background » There has been an increase in practice of performance measurement information but

Background » There has been an increase in practice of performance measurement information but use of this information… » There has been minimal research on performance measurement use at the federal level » It remains unclear whether performance measurement information is used for decision-making and management practices » This represents a gap in the literature on ways to improve and enhance performance measurement use in federal programs 2

Research Questions » Led to 3 overarching research questions: (1) What factors influence the

Research Questions » Led to 3 overarching research questions: (1) What factors influence the use of performance measurement information and how? (2) Under what circumstances is performance measurement information being used? (3) Who is using this information? 3

Significance » Has potential to positively influence decision and management practices » Helps to

Significance » Has potential to positively influence decision and management practices » Helps to inform policy, research and practice - Better equip federal managers and staff to use performance measurement information in a way that best meets their program needs 4

Theoretical Framework » Builds upon a performance measurement utilization model » Several uses to

Theoretical Framework » Builds upon a performance measurement utilization model » Several uses to consider in the decision making process + Instrumental + Non-instrumental » Factors influencing the use of performance measurement information + Rational/Technocratic Factors + Political/Culture Factors + Organizational Complexity 5

Hypotheses H 1 Use of performance measurement information consists of both instrumental and non-instrumental

Hypotheses H 1 Use of performance measurement information consists of both instrumental and non-instrumental use and each type of use is influenced by different factors. H 2 Organizations with increased complexity are more likely to use performance measurement information. H 3 Rational/technocratic factors are positively related with performance measurement use. H 4 Political/cultural factors are positively related with performance measurement use. H 4 -a Organizations that involve stakeholders in the performance measurement process are more likely to use performance measurement information. H 4 -b Organizations with high innovative and reward practices are more likely to use performance measurement information. 6

Research Design » Comparative case study » Mixed methods approach » Why two program

Research Design » Comparative case study » Mixed methods approach » Why two program offices? + Diversity in size, culture and structure + Both have performance measurement systems 7

Quantitative Methodology » 28 -item cross-sectional survey » Piloted in January 2012 » Population:

Quantitative Methodology » 28 -item cross-sectional survey » Piloted in January 2012 » Population: 326 managers and staff » Administered both web-based and paper-based between March 1, 2012 - May 1, 2012 Thank you for your participation in this survey. The purpose of this survey is to examine how and why federal programs use performance measurement information at the program level (not individual grantee or personnel performance level). This study is important to your program office and the field of public administration because it will provide a better understanding of factors that influence the use of performance measurement information. Your knowledge and expertise can provide valuable insight on this topic and your participation is greatly appreciated. 8

Quantitative Methodology » Dependent Variables ˃ Instrumental use: accountability, improvement ˃ Non-instrumental use: understanding,

Quantitative Methodology » Dependent Variables ˃ Instrumental use: accountability, improvement ˃ Non-instrumental use: understanding, mobilization » Independent Variables ˃ Organizational Complexity: levels of involvement, formulation and specialization ˃ Rational/Technocratic: budget, staffing, training and information ˃ Stakeholder Involvement: developing performance measurement, supporting change and hold office accountable ˃ Organizational Culture: innovative practices, rewarding managers/staff » Control Variables ˃ program office, position, length of time in Agency and program office 9

Quantitative Analyses Descriptive • • • N= 114 Managers= 29% Staff= 70% Length of

Quantitative Analyses Descriptive • • • N= 114 Managers= 29% Staff= 70% Length of time in Office= 85. 5% Overall program office distinctions: (1) Non-instrumental use for understanding (2) Rational/technocratic factors (3) Organizational culture Factor • Dependent variables 3 Constructs - Performance Measurement Use - Non-instrumental Use - Instrumental Use • Independent variables 4 Constructs - Organizational Complexity - Organizational Culture - Rational/Technocratic - Stakeholder Involvement 10

Multiple Regression Analyses Unstandardized B Coefficients Independent Variables Performance Measurement Use Non-instrumental Use Organizational

Multiple Regression Analyses Unstandardized B Coefficients Independent Variables Performance Measurement Use Non-instrumental Use Organizational Complexity . 259* -. 011 . 404** Organizational Culture . 207* . 227* . 055 Rational/Technocratic . 587*** . 412** . 419** . 080 . 094 . 014 Program Office 1 -. 184 -. 116 -. 147 Management . 180 . 257 -. 019 Time in Program Office . 155 . 065 . 073 Time in Agency . 248 . 141 . 301 R-squared=. 476 Significance=. 000 N= 114 R-squared=. 245 Significance=. 0001 N= 114 R-squared=. 341 Significance=. 000 N= 114 Stakeholder Involvement p<. 05*, p<. 01**, p<. 001*** 11 Instrumental Use

Quantitative Analyses Interviews • N= 20 • Length: 30 -60 minutes • Conducted 9/2012

Quantitative Analyses Interviews • N= 20 • Length: 30 -60 minutes • Conducted 9/2012 – 1/2013 Document Review • Data collection and review: 3/2012 to 2/2013 • Types of documents review included annual performance reports, • General Themes: program improvement; Congressional budget justifications, role of leadership; lack of resources; role funding announcements, and annual of stakeholders; structure of program reports office • Themes • Program Specific Themes: - Use and/or intended use of - Program Office 1: stakeholders role in performance measurement technical assistance; timely/accessible data; information for accountability, policies/procedures program improvement and - Program Office 2: need for understanding; challenges with developing performance measures; stakeholders role in performance measurement development 12

Qualitative Analysis Interview Quotes: “There are standard operational procedures for most of the work

Qualitative Analysis Interview Quotes: “There are standard operational procedures for most of the work performed by POs [project officers]. While many of these documents are “dense”, they provide a great foundation in order to execute tasks from start to finish” “Making performance information more readily accessible…. ” “It’s a relatively new system and … still identifying ways to help grantees actually utilize the performance data information versus just reporting it because it is a requirement” 13

Key Overall Findings » Instrumental use of performance measurement information was indicated by both

Key Overall Findings » Instrumental use of performance measurement information was indicated by both program offices » Rational/technocratic factors play a key role in the performance measurement use » Instrumental and non-instrumental use were influenced by both similar and different factors • Role of stakeholders in developing performance measures as well as providing technical assistance to grantee organizations • Structure of the program office appears to effect the use of performance measurement information 14

Implications » Communicating the value and importance of performance measurement information » Continuous assessment

Implications » Communicating the value and importance of performance measurement information » Continuous assessment of performance measurement resources » Cultivating and rewarding innovative practices for performance measurement use » Broadening the interpretation of performance measurement use to include both instrumental and noninstrumental use » Expanding research to be inclusive of both managers and staff and to assess use at the federal program level 15

Limitations » Low response rate » Self-reported data » Response bias » Organizational types

Limitations » Low response rate » Self-reported data » Response bias » Organizational types 16

Conclusion » Contributed to a limited body of knowledge on the use of performance

Conclusion » Contributed to a limited body of knowledge on the use of performance measurement information at the federal level » Added a unique perspective for examining the use of performance measurement information » Provided a greater understanding that may help determine ways to enhance the linkage between performance measurement use and management practices in federal agencies 17

Thank you! Julia Bryan, DPA, MPH, CHES juliabryan 76@yahoo. com 18

Thank you! Julia Bryan, DPA, MPH, CHES juliabryan [email protected] com 18