Air Force Institute of Technology Integrity Service Excellence

Air Force Institute of Technology Integrity - Service - Excellence Foundations for Measuring Effectiveness Major Richard ‘Kelly’ Bullock Dr Richard F. Deckro Department of Operational Sciences Air Force Institute of Technology Wright-Patterson AFB, OH USA 30 August 2005

Research Motivation n Accurately predicting outcomes and assessing progress has challenged leaders since the earliest use of national power to achieve political aims n n Many theories on how to employ national power to achieve desired endstates n n Especially true of military power Douhet, Jomini, Trenchard, Mitchell, Schelling, … Effects-based Operations n Not new, but efforts to ‘institutionalize’ EBO concepts are n US Joint doctrine and service doctrine, particularly USAF doctrine, has undergone change to reflect EBO concepts n History has shown theory is of little value if not supported by an empirically feasible measurement method One accurate measurement is worth a thousand expert opinions. – ADMIRAL GRACE HOPPER, 1906 – 1992 Integrity - Service - Excellence 2

Patton on MOEs… Discussion between General George Patton and General Orlando Ward during WWII: “How many officers did you lose today? ” asked Patton. “We were fortunate, ” Ward replied. “We didn’t lose any officers. ” “Goddamit, Ward, that’s not fortunate! That’s bad for the morale of the enlisted men. I want you to get more officers killed. ” A brief pause followed before Ward said, “You’re not serious, are you? ” “Yes, goddamit, I’m serious! I want you to put some officers out as observers, ” said Patton. “Keep them well up front until a couple get killed. It’s good for enlisted morale. ” Integrity - Service - Excellence 3

Overview n Background Measurement Concepts n Measurement Theory n n Application of Measurement Effects-based Operations n Effects n n Effectiveness Measurement Foundations Definitions n Concepts n Framework n n Takeaways Integrity - Service - Excellence 4

Background Measurement Concepts n Measurement is the objective representation of real-world objects, processes, and phenomenon n Measurement captures information through attributes n n A real-world system is defined by the attributes chosen to represent it n n Attributes may not be directly observable Measurement is an abstraction Measurement is the process of assigning symbols to a system attribute such that the assigned symbols reflect the underlying nature of the attributes Integrity - Service - Excellence 5

Background Measurement Concepts validity – how awell measure whatthe it was supposed to represent amplitude – well how a measure represents higher order constructs reliability – how consistent orreflects repeatable measurement process is System Measure Assessment Abstract Attributes Numbers Reasoning & Mathematics Empirical Object, Process, or Phenomenon Numerals Insights & Information = source for potential error random – ‘noise’ variation from any impacting the system observational – oversight of a construction key system attribute or source using the wrong measure for a system process attribute systemic – (bias) derives from of the between measure or definition of data the measurement Statistical Theory – concerns making inferences from Measurement Theory – link data and real-world To measure is to know. – LORD KELVIN, 1824 – 1907 Integrity - Service - Excellence 6

Background Measurement Theory n n Measurement Theory is a branch of applied mathematics that attempts to: n Describe, categorize, and evaluate the quality of measurements n Improve the usefulness, accuracy, and meaningfulness of measurements Representational view of measurement built upon 3 theorems Set Y Set X n E Representation – Proves the measure f preserves the relationships in X after mapping into Y (i. e. model validation) Uniqueness – Characterizes the family of measures that preserve the realworld relationships (i. e. scale type) 2. 5 B 8. 9 3. 1 5. 4 3. 7 C D n 1. 2 A measurement Nominal 6. 2 Ordinal Interval Ratio n Meaningfulness – Shows resultant is invariant to scale change for measures meeting the uniqueness condition Integrity - Service - Excellence Absolute 7

Background Application of Measurement n Application of measurement is… an art n the mechanism for extracting information from empirical observation n within a specified context n the translation of behavior into a set of ‘vital signs’ indicating variations in behavior n n Application guidelines n Frameworks n Vertical n Horizontal n Effectiveness vs. Efficiency n Measure Types & Characteristics . . . things are to you such as they appear to you and to me such as they appear to me. . . – PROTAGORAS, 485 – 421 B. C. Integrity - Service - Excellence 8

Background Frameworks Measure of Effectiveness (MOE) Objective 31 Effectiveness Vertical (or Hierarchical) Structure What? Fundamental Objective Sub. Objective O 1 Why? Sub. Objective O 2 Value V 1 Value V 2 Sub. Objective OX Value VY Objective 21 Objective 32 Objective 11 Objective 22 Objective 33 Objective 12 Objective 23 Objective 34 Objective 1 M Objective 24 Objective 35 Objective 2 N Objective 36 Objective 3 P How? Horizontal Structure Attribute 1 Attribute 2 Attribute. Z Efficiency Measure of Performance (MOP) We must make the important measurable, not the measurable important. – ROBERT MCNAMARA, 1916 – Integrity - Service - Excellence 9

Background Measure Types & Characteristics Natural Direct Proxy Constructed - Commonly understood measures directly linked to fundamental objective - Example: Profit - Measures directly linked to the fundamental objective but developed for a specific purpose - Example: Gymnastics scoring - In general use measures focused on an objective correlated with the fundamental objective - Example: GNP (economic well being) - Measures developed for a specific purpose focused on an objective correlated to the strategic objective - Example: Student grades Strategically-linked – traceable to fundamental objective; responsive to change and provides indication of how much change can be attributed to an input Timely – able to be collected and processed in a timeframe needed to be relevant within the context Objective – 1) Collection: easy to understand, same regardless of the assessor, same under similar circumstances; ‘face-value’ or whether the measure logically represents what it is supposed to represent 2) Interpretation: unambiguous interpretation; distinguish between desired and undesired consequences Economical – Collection and processing should provide benefits that off-set the burden of measurement activities Complete – Measures should address all areas of concern in enough detail to discern reasons for differences in actual and expected system results Measurable – hold for Measurement Theory conditions; within a given context if the measure can be feasibly obtained with available resources (i. e. operational ) Integrity - Service - Excellence 10

Background EBO Overview n Effects-based Operations (EBO) n Theory for the employment of capabilities in dynamic and uncertain environments in a manner to best attain objectives n Provides a conceptual framework for determining the integration and application of capabilities to achieve specific effects to influence the environment of interest yielding desired outcomes “You begin with the objectives that indicate what you really care about. Then you follow simple logical reasoning processes to identify the mechanisms by which the objectives can be achieved. Finally, for each mechanism, you create alternatives by asking what control you have over that mechanism. ” Keeney, Ralph, Value Focused Thinking: A Path to Creative Decision Making, Harvard University Press, Cambridge, 1992. n Key tenets Focus on end outcomes n Reduced emphasis on weapon systems n De-emphasis on destruction as a sole means of achieving effects n Not everything that can be counted counts, and not everything that counts can be counted. – ALBERT EINSTEIN, 1879 – 1955 Integrity - Service - Excellence 11

Background EBO Overview measures measures n measures measures Assessment Planning measures Employment Effects-based Operations Measures Effectiveness –ifend-state; tie provide feedback on if Assessment –– determining thepillars intended effects were achieved and Employment efficient employment oftogether; capabilities; increased Planning –offocus on desired develop strategy (who, emphasis what, strategy progress towards desired end-state they arewhy, shaping the desired outcomes on non-lethal means where, when, and how) Integrity - Service - Excellence 12

Background Effects Overview n Effect n Consequence System state change or result of a particular action Tactical Input Effect Input Direct Parallel Intended Positive Physical Effect Indirect Parallel Intended Positive Functional Input Effect Direct Parallel Intended Positive Functional Effect Indirect Unintended Negative Psychological Input Input Effect Input System Boundary Effect Direct Parallel Intended Negative Psychological Direct Parallel Intended Positive Psychological Direct Sequential Intended Positive Functional Strategic Operational Input Attribute Effect Indirect Sequential Cascading Intended Positive Functional Indirect Unintended Positive Functional Indirect Sequential Cascading Intended Positive Functional Indirect Sequential Cascading Unintended Positive Functional Direct (First-order) Indirect (Higher-order) Indirect Parallel Cumulative Timing Effect Intended Positive Systemic Impact Indirect Unintended Negative Effect Persistence Psychological Indirect Sequential Domain Cumulative Effect Intended Positive Systemic Indirect Cascading Unintended Negative Psychological Parallel Sequential Cascading Cumulative Indirect Parallel Intended Intent Effect Positive Un-intended (Collateral) Functional Result Indirect Parallel Intended Positive Psychological Effect Order Types Level. Indirect Effect Integrity - Service - Excellence Unintended Negative Psychological Positive Negative Permanent Temporal System Behavior Physical Functional Systemic Psychological Tactical Mechanism Operational Strategic 13

Approach Definitions DEFINITION 1: A SYSTEM is a set of elements where relationships exist between the elements and the SYSTEM has a purpose or normative behavior. DEFINITION 2: A system ELEMENT, or SUBSYSTEM, is a system providing functionality or support to a parent system. DEFINITION 3: A MODEL is a formal image of an empirical structure. DEFINITION 4: An ATTRIBUTE, or NODE, is a characteristic, feature, or property of a system that is directly or indirectly observable. DEFINITION 5: A MEASURE is a model of an attribute. DEFINITION 6: A MEASUREMENT, or observation, is a particular manifestation, or instantiation, of an attribute. DEFINITION 7: A system STATE is a particular instantiation of all system attributes, or state variables, at a particular point in time. DEFINITION 8: An EFFECT is a system state change. DEFINITION 9: EFFECTIVENESS gauges the magnitude of a system state change. DEFINITION 10: An END-STATE characterizes the desired measurements for all system attributes, or state variables. The beginning of wisdom is calling things by their right names. – CONFUCIUS, 551 – 479 B. C. Integrity - Service - Excellence 14

Approach Product Structure Development Process Empirical System 1. System Identification • Determines system boundary • Covers all pertinent aspects of desired end-state 2. Sub-system Identification • Decomposes into smaller parts • Only relevant sub-systems are needed 3. Sub-system Importance • Discerns relative importance among selected sub-systems • Amounts to weighting each sub-system 4. Attribute (Node) Identification • Characterizes each sub-system by its salient features • Only attributes relevant to context are needed 5. Attribute (Node) Importance • Discerns relative importance among selected attributes (nodes) • Amounts to weighting each attribute (node) Attribute 1 6. Measure Development • Quantification of attributes (nodes) • Constructed measures may be needed Measure 1 • Measures should hold for metric properties Formal System (model or product structure) System Sub-system 1 sub-system weight Sub-systemn Attributem attribute weight =1 Measure q m measure =1 Integrity - Service - Excellence 15 =1

Approach Concepts Time t Empirical System A = < a. A 1, …, an > t=0 Element (Subsystem) ai t=T Effect (of inputt = 0 at t = T) t = TEnd-state Attribute (Node) Link Input Mechanism System Elements Formal (Subsystems) System Attributes (or Model) (Nodes) x. A = < x 1, xx xn-1, xn > 2, …, A Measures System Measurement x. A x 1 x 2 1 ά 1 xn-1 2 ά 2 x. A xn n-1 άn-1 x 2 m άm 1 ά 1 xn-1 2 ά 2 x 1 xn n-1 άn-1 1 m άm x 2 ά 1 xn-1 2 ά 2 xn n-1 άn-1 m άm Measurements (Observations) x. A = < 0, 0, …, 0, 0 > x. A = <. 2, . 7, …, . 3, . 5 > x. A = < 1, 1, …, 1, 1 > Effectiveness (of inputt = 0 at t = T) Count what is countable, measure what is measurable, and what is not measurable, make measurable. . . – GALILEO GALILEI, 1564 – 1642 Integrity - Service - Excellence 16

Takeaways n Measurement turns real-world system behavior into a set of ‘vital signs’ for the purpose of monitoring the system n Measurement Theory provides a robust basis for turning qualitative observations into quantitative data n Effectiveness vs. Efficiency Measure of Effectiveness (MOE) n External measure n Invariant to means of achievement n Are we doing the right things? n Measure of Performance (MOP) n Internal measure n Coupled to means of achievement n Are we doing things right? n n EBO is an instantiation of an existing, structured, decision-theoretic framework (VFT) for approaching problems with abstract or ill-defined objectives n An effect is a system state change n Effectiveness gauges the magnitude of the change Integrity - Service - Excellence 17

Questions & Discussion A prudent question is one-half of wisdom. – SIR FRANCIS BACON, 1561 – 1626 Integrity - Service - Excellence 18