EVALUATIO N SECTION B Three Components 2 Measuring

EVALUATIO N SECTION B

Three Components 2 Measuring Performance Process vs. Results-based Evaluation Building a Toolkit

3 How does your Board know a) Whether the organization is doing the right things? b) Whether the organization is having any impact on public safety/welfare? c) Whether the organization is well-managed?

Measuring Performance 4 bother? ? . . . . Why

So what about. . 5 To - demonstrate accountability - provide evidence for decisionmaking - take the governance role seriously - point to opportunities to improve

6 To - provide reassurance to stakeholders - inform future planning and development - compare performance against other similar organizations (norm referenced eval) - assist with evidence based public reporting

Simply put. . . 7 Self reflection is not useful without data

8 Why not role model the behavior we expect of licensees? ?

Measurement 9 for the sake of measurement is a waste of energy

10 So. . what should be the focus?

Consider mandate 11 what are your objects as stated in legislation? what is your purpose? why do you exist?

Consider legislative aims 12 What else was the legislation contemplating? (quality assurance; zero tolerance of sexual abuse; transparency; etc. )

Consider legislative targets 13 What about specific expectations set in legislation – notice periods; appeals; decision time periods etc?

Consider governance risk 14 Perhaps consider corporate governance requirements Perhaps focus on the Boards’ conduct Perhaps your orientation program Perhaps whether your strategic plan is making progress

Consider organization risk 15 What about finances? What about licensing exams? What about the building? What about staff health? What about reputation? What about political interference?

16 What does the Board want to know? In planning, think about the questions of the Board.

17 What does the public want to know? In planning think about who is your public and what might that public want to know.

Creating Alignment 18 Make sure your questions align with your measurement strategy!

Process Evaluation vs. Results-based Evaluation 19 Considering the how – Understanding our options

Process vs. Results 20

21 Efficiency vs. Effectiveness

22 Creating a mindset and a culture of evaluation

23 Varied roles in measurement Board? CEO?

Board 24 • • Adopting an evaluation framework Identifying areas to be measured Utilizing the tools chosen Asking probing questions

CEO 25 • • • Proposing and championing an evaluation framework Identifying areas to be measured Implementing the framework Educating staff on use of the data Supporting the Board’s use of the data Answering questions

Process Measures 26 Board meeting evaluation Committee meeting evaluation Annual performance reviews Satisfaction surveys Decision tree checklists Questionnaires Others. . .

Results Measures 27 Balanced Scorecard Legal audit Internal self audit Financial audit Outcomes evaluation tools (ie Program Logic) Benchmarking tools Others. . .

28 Setting key indicators or outcomes

Remember. . . 29 The measurement focus should be on mandate and strategy

Exercise 30 Using your own legislation take the next 10 minutes to identify two or three items for possible measurement

Building a Toolkit 31

Options 32

Process Tools 33 Surveys Opinion gathering Themes to consider Testing ideas

Example 34 Practice Advisory Service Satisfaction Survey

Board & Committee function evaluations 35 Meeting flow Preparation Decision making Sufficient information Conduct of members

Example 36

Example 37

Program evaluation 38 Can be both process and outcomes oriented

Example 39 Program Logic Model

Example 40

Results-based 41 Balanced Scorecard Use of indicators Outcomes oriented Linked to mission and mandate Regular reporting tool

Sample Dashboard 42 Scorecard Dashboard and Indicators Report

Sample indicators 43 Appeals # of complaints decisions upheld/ # of appeal decisions received Case Disposition # of cases closed within established standard for case type/# of cases closed in quarter Risk Incidents # of risk incidents identified in quarter

Setting Targets 44

Legal Audits 45 Third party Objective Alignment with legislation

Small “r” research 46 Adds rigor to decision-making Tests hypotheses Increases the science of regulation

Accreditation 47 Examples NCCA, ANSI, ISO, SCC Third party Objective Benchmarking External recognition

Others? 48

Evaluation Framework 49 Provides deliberately identified objectives and tools Establishes the story you want to tell

Getting Started 50 Just start Identify champions Recognize measurement is imperfect Keep an open mind Be prepared to learn

Involve partners 51

Evaluate Annually 52

53 Measurement and Evaluation is iterative and will never end

Exercise 54