The SciencePolicy Interface in Logicbased Evaluation of Forest

The Science/Policy Interface in Logic‑based Evaluation of Forest Ecosystem Sustainability Keith M. Reynolds, USDA Forest Service K. Norman Johnson, Oregon State University Sean N. Gordon, Oregon State University USDA Forest Service PNW Research Station

Acknowledgments Ø USDA Forest Service Washington Office ü Ø National Forest System, Ecosystem Management Pacific Northwest Research Station ü Human and Natural Resource Interactions RD&A Program USDA Forest Service PNW Research Station

Objectives 1. Illustrate the value of a logic-based approach in designing a formal specification to evaluate the Montreal criteria and indicators. 2. Identify the roles of science and policy in this effort. 3. Highlight lessons learned from this process. 4. Suggest some general recommendations. USDA Forest Service PNW Research Station

Overview Ø Ø Ø Introduction Knowledge bases and logic modeling Analysis Model design issues Lessons learned Recommendations USDA Forest Service PNW Research Station

Introduction: criterion Ø A standard that a thing is judged by (Prabhu et al. 2001). ü Ø Ø Criteria are the intermediate points to which the information provided by the indicators can be integrated and where an interpretable assessment crystallizes. Principles [e. g. , sustainability] form the final point of integration. A criterion should be treated as a reflection of knowledge. ü It can be viewed as a large‑scale selective combination … of related pieces of information. USDA Forest Service PNW Research Station

Introduction: indicators An indicator is any variable or component of the forest ecosystem … used to infer attributes of the sustainability of the resource and its utilization. Indicators should convey a ‘single meaningful message. ’ This ‘single message’ is termed information. It represents an aggregate of one or more data elements with certain established relationships (Prabhu et al. 2001). USDA Forest Service PNW Research Station

Introduction: measurement endpoint Ø Some indicators are simple. ü Ø Their definition suggests an obvious one‑to‑one correspondence between an indicator and a metric for that indicator. Definitions of some indicators are more complex. ü They represent a synthesis of two or more data elements, which we refer to as measurement endpoints. USDA Forest Service PNW Research Station

Introduction: scales of application Ø Ø Purpose varies with scale (Castañeda 2001) National and regional ü ü Ø Policy instruments to evaluate laws, policy, regulations E. g. , Montreal Process, NWFP, ICBEMP Management unit ü ü Evaluation and adjustment of management practices E. g. , CIFOR, USDA FS IMI (LUCID) USDA Forest Service PNW Research Station

Knowledge bases Ø A form of meta database ü ü ü Ø A formal logical representation of how to evaluate information Networks of interrelated topics Mental map Advantages ü ü Interactive, graphic design (modularity) Numerous & diverse topics can be analyzed within a single integrated analysis USDA Forest Service PNW Research Station

Knowledge bases: forms of uncertainty Ø Probabilistic uncertainty ü Ø Uncertainty of events Linguistic uncertainty ü Uncertainty about the definition of events § ü ü Vagueness or imprecision A proposition is the smallest unit of thought to which one can assign a measure of truth (strength of evidence) SE: a measure that quantifies the degree of support for a proposition provided by its premises USDA Forest Service PNW Research Station

Knowledge bases: networks of topics = topic Concern 1 Concern 2 Ecostate A Ecostate B Etc. Ecostate C Data link Ecostate D Data link Data

Knowledge bases: topics Ø Ø Each topic typically evaluates a proposition Attributes of topics ü ü ü Name (e. g. , watershed condition) Proposition (e. g. , WS condition is suitable) Strength of evidence for the proposition § ü ü Net. Weaver scale is [-1, 1] A formal logic specification (e. g. , a proof) Documentation § Explanation, source, citations, assumptions USDA Forest Service PNW Research Station

Knowledge bases: evaluation Ecostate A Ecostate B Data link Evaluate data Get data requirements Concern 1 Ecostate C Data link

Knowledge bases: strength of evidence USDA Forest Service PNW Research Station

Analysis: Montreal C&I Ø Ø The Montreal specifications provide relatively clear definitions of biophysical, socioeconomic, and framework attributes requiring evaluation (WGCICSMTBF 1995). . . But, design of evaluation procedures that allow interpretation of the Montreal C&I is one of the major technical issues that remain to be resolved (Raison et al. 2001). USDA Forest Service PNW Research Station

Analysis: conceptual framework 1. Specified conditions or outcomes to be sustained (the indicators). 2. A measure for each condition or outcome. 3. Calculation of the level of the indicator over some time period using the selected measure. 4. A frame of reference for gauging sustainability. 5. Methods for evaluating sustainability (sustainability check). 6. A monitoring program. 7. A formalism that supports requirements 1 to 6. USDA Forest Service PNW Research Station

Analysis: logic models as design frameworks Ø Ø Logic models (knowledge bases) provide a formal specification for organizing and interpreting information. Net. Weaver kb developer system ü ü ü Problem represented in terms of propositions about topics of interest and their interdependencies. Topics translated into propositions. Lexical uncertainty. USDA Forest Service PNW Research Station

Analysis: logic models as design frameworks (continued) Ø Need for transparency (Prabhu et al. 2001) ü ü Ø Model development ü Ø Models embody important policy decisions. Models depend on value judgments and critical assumptions that need clear documentation. Graphic representation is an effective basis for organizing discussion and for evolution of design. Communication ü ü Between scientists and policy makers. With interested publics. USDA Forest Service PNW Research Station

Design issues: model organization Basic organization of topics. For example, evaluation of criteria in the current prototype.

Design issues: model organization An alternative organization with very different emphasis on criteria.

Design issues: synthesis Ø Ø AND operator: arguments evaluated as limiting factors. SUM operator: arguments contribute incrementally to evaluation and can compensate.

Design issues: synthesis Another example, including the OR operator.

Design issues: weighting Ø Intrinsic weights ü ü ü Ø Each topic in a Net. Weaver logic model has an intrinsic weight attribute. E. g. , set weight attribute on any topic to adjust its contribution of evidence to a proposition. Bad idea: part of specification, but not obvious. Explicit weights ü ü Better, but add another layer of subjectivity. Some valid purposes, however. USDA Forest Service PNW Research Station

Design issues: reference conditions Ø Ø Each elementary network evaluates a measurement endpoint against reference conditions. Lack of reference conditions is a basic problem for most measurement endpoints.

Design issues: reference conditions Implementation of a elementary network to evaluate measurement endpoint against reference conditions.

Design issues: qualitative measures Outcomes evaluated on an ordinal scale.

Design issues: reliability of data Ø Reliability of data for evaluation of Montreal C&I. ü Ø Formal representation of stochastic uncertainty is problematic in the context of a logic model. ü Ø Stochastic, rather than lexical, uncertainty Not addressed in the current Montreal C&I prototype. Possible solution ü ü Adjusting topic weights with a normalized metric such as standard error of the mean. Problems: availability, unknown error correlations USDA Forest Service PNW Research Station

Design issues: precision of knowledge Sequential OR (SOR) to specify multiple alternative pathways in order of preference.

Lessons learned 1. Lexical uncertainty is an important issue in evaluation of Montreal criteria and indicators. 2. Many aspects of evaluating sustainability cannot be answered by science alone. 3. Acquiring data on sustainability is necessary but not sufficient for setting policy. 4. Evaluating sustainability is not the same as defining desired future conditions. 5. Evaluating the state of sustainability and deciding how to respond are separate but interdependent decision processes. USDA Forest Service PNW Research Station

Recommendations Ø Ø Ø Assess the policy role in sustainability evaluation, and undertake a policy review of model organization and strategies for integrating sustainability information. The clearest, and most critical, role of science is in development of reference conditions. A major effort is needed to identify measurement endpoints for indicators of the institutional framework (criterion 7). USDA Forest Service PNW Research Station

Followup Ø Scientist peer review (Mar 14/21, 2002) ü ü ü John Gordon (forester, Yale) Jerry Franklin (ecologist, U. of Washington) Hal Salwasser (Dean, Co. F, Oregon State University) Denise Lach (sociologist, Oregon State University) Gordie Reeves (fisheries biologist, PNW) Richard Haynes (economist, PNW) USDA Forest Service PNW Research Station

Followup Ø Management peer review (Mar 14/21, 2002) ü ü ü Darrel Kenops (supervisor, Willamette NF) Gloria Brown (supervisor, Siuslaw NF) Ted Lorensen (manager, Oregon Do. F) Jon Martin (REO monitoring, Region 6) Dick Phillips (economist, Region 6) Sarah Crim (analyst, Region 6) USDA Forest Service PNW Research Station

Followup Ø Demo with criterion 6 data from 2003 report ü ü Ø Richard Haynes (PNW) Ken Skog (FPL) Sue Alexander (PNW) Others? Demo with LUCID link? ü Possibly July/Augest 2002 USDA Forest Service PNW Research Station

Authors Ø Keith M. Reynolds ü ü Ø K. Norman Johnson ü ü Ø USDA Forest Service, Pacific Northwest Research Stn. kreynolds@fs. fed. us Oregon State University, College of Forestry norm. johnson@orst. edu Sean N. Gordon ü ü Oregon State University, College of Forestry sean. gordon@orst. edu USDA Forest Service PNW Research Station