USEPA REGION 4 METHOD PERFORMANCE AND DATA COMPARABILITY

USEPA REGION 4 METHOD PERFORMANCE AND DATA COMPARABILITY PILOT PROJECT L. Huff 1, 2, R. Frydenborg 2, 4, M. Brossett 2, 5, C. Jones 2, 5, T. Pugh 2, 5, M. Vogel 2, 6, A. Dossett 2, 7, B. Crouch 2, 8, J. Glover 2, 9, D. Arnwine 2, 10, L. Dorn 2, 11, C. Decker 2, 11, J. Harrison 2, 12, D. Melgaard 2, 12, L. Gabanski 13, J. Diamond 3, 14, S. Stribling 3, 14, J. Gerritsen 3, 14 1 AL Majors Cr (Baldwin Co. , AL) R 4 1996 Joint Bioassessment Exercise Dept of Environmental Management; 1400 Coliseum Boulevard, PO Box 301463, Montgomery, AL 36130 -1463; e mail: esh@adem. state. al. us; 2 R 4 Workgroup Member; 3 Technical Support; 4 FL Dept. of Environmental Protection; 5 GA Dept. of Natural Resources; 6 KY Dept of Environmental Protection; 7 MS Dept of Environmental Quality; 8 NC Dept. of Environment and Natural Resources; 9 SC Dept. of Health and Environmental Control; 10 TN Dept. of Environment and Conservation; 11 R 4 EPA-Athens; 12 R 4 EPA-Atlanta; 13 EPA Headquarters; 14 Tetra Tech, Inc. Introduction • The water quality program of each EPA Region 4 state includes biological monitoring as an integral part of its assessment of water quality. • Documenting method performance characteristics is essential as states struggle to develop biological indicators that link nutrient enrichment, sedimentation, and habitat degradation to biological community response. Method Performance Measures The project focuses on performance characteristics of each agency’s final assessments based on their established protocol. Four performance characteristics will be measured: precision, sensitivity, bias, and responsiveness. Several methods of measuring these characteristics will be evaluated. • EPA conducted the National Wadeable Stream Assessment (WSA) using a single biological assessment protocol at over 500 sites. The WSA provides an estimate of water quality on a national scale, but results may not agree with state results. Documentation of method performance characteristics may provide EPA with a consistent measure to determine what data should be used to assess water quality, concentrate resources into improving state methods, and help understand why state assessments differ. • There is currently no guidance for developing and documenting performance characteristics and comparability of bioassessment methods. We are conducting a region-wide pilot project to: • Propose a framework for states to calculate and report data quality and performance characteristics of macroinvertebrate bioassessment methods. • Review, test, and revise agreed-upon methods Focus Region 45 65 66 71 75 NC SC AL EPA FL MS NC TN KY TN FL GA Site Classification 1 L 3 B B B B L 4 Focus Area 2 AA AA 65 d/i AA AA 66 d/e/g 71 a/b/c/ e/f/g/h 71 e & 71 f/g/h 75 f/e 75 f Sampling Period 3 1 -12 6 -9 5 -6 1 -12 12 -2 1 -12 5 -9 1 -12 9 -2 Reference 4 E E E 2006 E E E E 2006 Test 4 E E E 2006 E E E E 2006 2006 E 2006 Duplicate 4 JBs 5 3 in NC & 3 in SC 3 in AL 3 in MS 3 in NC & 3 in TN 3 in KY & in TN 3 5 in FL 1) B: Bioregion; L 3: Level 3 Ecoregion; L 4: Level 4 Ecoregion; 2) Bioregion or L 4 in which project will be focused; AA: All available; 3) Months for which macroinvertebrate bioassessment criteria developed; 4) E: existing data; 2006: data collected during 2006; 5) JB: Joint Bioassessments – 3 -6 conducted at a gradient of good, fair, and poor sites. Bias, sensitivity, and responsiveness cannot be measured without independent assessment of the degree of impairment at each bioassessment site. A critical element of this project is the development of a regional Generalized Stressor Gradient (RGSG). R 4 Joint Bioassessments Data used to calculate and document Method Performance • 5 pairs of duplicate samples along stressor gradient • 10 reference reach samples within a site class • 10 “test” reach samples along a stressor gradient • Independent assessment of water quality at each site Table 1. Summary of project areas, data sources, and joint bioassessments. Development of a Regionalized General Stressor Gradient and Independent Assessments of Water Quality RGSG Factors: Quantifying Watershed Stress Landscape Level: • 2000 MRLC % Land Cover • % Urban, % Agriculture, and Road Density • Absolute Scale: Gradient thresholds set based on literature Habitat Alteration: • Habitat assessment data from each state • Agencies use variations of the RR/GP forms in EPA’s RBP manual (Barbour et al. 1999) • Relative Scale: Gradient thresholds based on “best available” in region. Chemical Pollution: • Water quality data collected by each agency • Differences in parameters collected, when data collected, and number of site visits • Most states have conductivity and nutrient data (NO 3/NO 2 -N, NH 3 -N, TP) • Relative Scale: Gradient thresholds based on “best available” in region. • Documentation of data quality objectives and method performance characteristics will help R 4 states to objectively determine how bioassessment data produced by their own and outside agencies should be used in management decisions. • Region 4 includes 14 Level 3 Ecoregions and 99 Level 4 Ecoregions, none of which are present in all eight states. The project will be restricted to bioregions shared by 2 or more states, with existing data or where data can be collected during 2006. • A survey conducted by ADEM was used to integrate the R 4 MPDC Pilot Project into each state’s baseline monitoring activities and to use existing data where available. Data used to measure method performance will Level 3 and 4 Ecoregions of the include at least: southeastern US Sougahatchee Cr (Lee Co. , AL) R 4 2004 Joint Bioassessment Exercise Quality Control Measures Table 2. Quality control measures of specific steps in protocol to be calculated for each agency. Quality Control Measures Definition Calculation Sorting and Subsampling Bias: Percent Sorting Efficiency (PSE) A second sorter checks for missed specimens PSE=(A/(A+B))*100 Where A=# of organisms found by 1 st sorter; and, B= # orgs. missed by 1 st sorter A 2 nd taxonomist rechecks organism counts PDE=([n 1 -n 2]/(n 1+n 2))*100 Where n 1= # of orgs counted by 1 st tax; and, n 2= # of orgs counted by 2 nd tax A 2 nd taxonomist re-identifies organisms PTD=[1 -(comppos/N)]*100 Where comppos= # of agreements; and, N=total # organisms in lger of two counts. Taxonomy Precision: % difference in enumeration (PDE) Precision: % taxonomic disagreement (PTD) If performance characteristics do not meet data quality requirements, this information may help each agency determine how they could potentially address the problem by identifying those steps that are most variable. If our assessments do differ, this information, in conjunction with the meta data associated with each of the reference and test samples, may help us determine why. We assume that methods with similar performance will provide similar assessments of water quality (Barbour et al. 1999). With the RGSG, combined with results of side-by-side bioassessments, it may be possible to : 1) evaluate the use of performance characteristics as a measure of data comparability, and 2) determine what other information is needed to accurately evaluate data comparability. Six “mini” joint bioassessment workshops will be conducted among subsets of the nine participating agencies (Table 1). Each agency is conducting 3 -6 joint bioassessments with another agency that shares the same bioregion or ecoregion. Method Performance Documentation Form and Database • We are developing forms for submitting bioassessment data to the state. • Entities will provide the following tables for reference, duplicate, and “test” sites: 1. Site description 2. Reference Reach Criteria 3. Bioassessment data 4. Ancillary data • An ACCESS database will be created to house all R 4 method performance data, evaluate the data in terms of site class, sampling period, and overall dataset, and to calculate and report performance characteristics.