Methods and Data Comparability Board the Board never
Methods and Data Comparability Board (“the Board”-- never “Bored”) Board Co-Chairs Herb Brass, USEPA & Charlie Peters, USGS October 11, 2001 (DRAFT 5) MDCB overview 1
How did there come to be a “Methods Board”? 1992 -- ITFM formed to respond to US 1996 --ITFM final report, recommending the OMB challenge to review and evaluate formation of a permanent Monitoring Council national water quality monitoring and Methods Board activities & develop of recommendations for improvement 1997 --NWQMC and MDCB chartered under Federal Advisory Committee Act (FACA). Advisory Committee on Water Information (ACWI) is the parent organization True collaboration among programs is possible if there is both the technical and the institutional framework to promote data comparability to assure data of known quality October 11, 2001 (DRAFT 5) MDCB overview 2
The Board is. . . A partnership of water quality experts from federal agencies, States, Tribes, municipalities, industry, and private organizations – 5 delegates, 5 alternates from each of three sectors – Workgroup members October 11, 2001 (DRAFT 5) l a er d e F Sta te ba /Tri l Other monitorin g interests MDCB overview 3
How we work… – Meet 3 -4 times a year, locations throughout U. S. – Each meeting lasts 2 -3 days; some people participate in person, others via phone – Workgroup conference calls as needed throughout year – Collegial, informal, and productive! October 11, 2001 (DRAFT 5) MDCB overview 4
How we work…caught in the act! October 11, 2001 (DRAFT 5) MDCB overview 5
Mission of the Board Create a framework for collaboration and comparability among programs by. . . identifying, examining, and recommending monitoring approaches that facilitate collaboration and yield comparable data and assessment results October 11, 2001 (DRAFT 5) MDCB overview 6
The Board’s guiding principle Deliver products in the short term while thinking and planning strategically in the long term October 11, 2001 (DRAFT 5) MDCB overview OUTR COMM EACH & UNICA TION COORD & CON INATION SENSU S GUIDE LINES PILOTS 7
Accomplishments “across the Board” iplinary c s i d r e t n i f its Because o oard has B e h t , p i ridge b membersh o t y t i n opportu n a d e d i v eatment r t pro r e t a w tween ion and t the gap be p m u s n o c ell as w s for human a , s e c r resour e t a w l a r n and u a m nat u h n e twe the gap be ealth. h l a c i g o l o ec October 11, 2001 (DRAFT 5) MDCB overview 8
Challenges “across the Board” Members are volunteers – not always able to participate or respond in a timely manner ed Limit -- of es c r u o res nds i k l l a October 11, 2001 (DRAFT 5) Developing and implementing outreach and communication strategies MDCB overview Need fo r ground rules on how operate to under consens us 9
The Board’s Framework for Comparability There are 4 critical elements of the framework for methods and data comparability October 11, 2001 (DRAFT 5) MDCB overview 10
The Board’s Framework for Comparability The Board’s workgroups develop and produce the projects/ products needed to address the key elements October 11, 2001 (DRAFT 5) MDCB overview 11
Getting to comparability – The 4 elements are the steps or building blocks moving us toward the goal of comparability. – Each of the Board’s workgroups is focused in one or more of these elements or steps – Effective and innovative outreach is an overarching Data need for each workgroup Reporting Lab Performance NEMI * PBMS * Accreditation * WQDE * Biology * Nutrients Field Performance DQOs & MQOs October 11, 2001 (DRAFT 5) MDCB overview 12
But wait…what about the impact of new technologies? different methods used at different times by different programs with different DQOs/MQOs… October 11, 2001 (DRAFT 5) MDCB overview 13
Why are new technologies of interest to the Board? Improve data quality – (increase specificity, sensitivity, precision, accuracy) Reduce cost of sampling and analysis – (materials for sample collection and analysis; time in field; time of sample preparation and analysis; time to report results; labor) – increase data quantity (representativeness, completeness) improve data quality Bottom line: potential to improve protection of ecological and human health October 11, 2001 (DRAFT 5) MDCB overview 14
The relationship of new technologies to the framework October 11, 2001 (DRAFT 5) MDCB overview 15
partnerships How do we achieve comparability in the face of new technologies? October 11, 2001 (DRAFT 5) collaboration MDCB overview 16
Role of the Board with regard to new technologies – Clearinghouse for analytical methods (NEMI) • to recognize contributors, facilitate technology transfer, and support data comparability (without endorsement of specific methods) – identify needs for new or improved monitoring techniques – group and prioritize methods – develop and promote guidelines to ensure methods and data comparability for priority methods October 11, 2001 (DRAFT 5) MDCB overview 17
The Board’s involvement with new technologies To date – Biology Workgroup – NEMI Phase I Future – Biology Workgroup – Nutrients Workgroup (NOAA, NIPH) – NEMI Phase II General – – PBMS Accreditation Federal Labs (ACFLAB) Outreach Water Quality Data Elements (WQDE) October 11, 2001 (DRAFT 5) MDCB overview 18
Where are we going and how are we going to get there. . . develop and deliver products in the short term while thinking and planning strategically in the long term. . . Data Reporting Lab Performance Field Performance DQOs & MQOs October 11, 2001 (DRAFT 5) MDCB overview 19
National Environmental Methods Index (NEMI) Workgroup Workgro up Chair Larry Ke ith, Insta nt Referenc e Source s (c ontract s upport) October 11, 2001 (DRAFT 5) MDCB overview 20
Mission of the NEMI Workgroup Data Reporting Lab Performance Field Performance DQOs & MQOs October 11, 2001 (DRAFT 5) To allow rapid communication and comparison of critical parameters of methods for use with methods selection and/or methods modifications and data comparability MDCB overview 21
What is NEMI? chemical physical biological NEMI is a web-based, searchable compendium containing method summaries of field and lab protocols microbiological October 11, 2001 (DRAFT 5) radiochemical MDCB overview 22
NEMI’s relationship to the framework – NEMI database ensures that the consideration of field and measurement methods plays an active role in the planning and implementation phases of a program – NEMI data fields include detection levels, bias, precision, and other QA/QC requirements necessary for documenting and reporting on data quality October 11, 2001 (DRAFT 5) MDCB overview Data Reporting Lab Performance Field Performance DQOs & MQOs This information, in a like format, will make methods comparability for interagency use, as well as for use by the general public, quick and cost effective. 23
NEMI Workgroup members Data Reporting Lab Performance FEDERAL – DOE – USEPA – USGS LOCAL/STATE/REGIONAL – DE River Basin Co – East Bay Municipal Utility District – NYS Department of Health Field Performance DQOs & MQOs INSTITUTES/ORGANIZATIONS – – – AOAC International Instant Reference Sources (contract support) Standard Methods ASTM SUNY Albany, School of Public Health BUSINESS/INDUSTRY – – It is critical to involve as many method entities as possible in the NEMI building process to insure adequate representation of available methods Merck & Co. , Inc. Montgomery Watson Tetra Tech (contract support) Dyn. Corp (contract support) October 11, 2001 (DRAFT 5) MDCB overview 24
NEMI Workgroup accomplishments – Produced Phase I prototype of web-based methods database with over 100 different methods – Developed business rules outlining how methods are summarized and catalogued – Continued Phase II development – Created high interest levels among many agencies– cited in recent GAO report – Presenting a paper on NEMI at the AWWA meeting in November, 2001. October 11, 2001 (DRAFT 5) MDCB overview 25
NEMI Workgroup challenges Getting d metho to s e c r u so submit in s d o h t me t a m r o f NEMI Obtaining methods from different organizations Resolvin g proprieta ry/ copyrigh t issues f or some me thods October 11, 2001 (DRAFT 5) MDCB overview 26
Where are we going? – Complete Phase II NEMI with more methods (approximately 350 methods--chemical, immunoassay, microbiological, radiological), and provide public access – Initiate Phase III NEMI with more methods including field and biological methods – Expand NEMI to other matrices (e. g. , soil) October 11, 2001 (DRAFT 5) MDCB overview 27
How do we get there? – Work proactively with organizations that publish methods to encourage them to submit methods to NEMI. – Find other agencies to work with us to contribute funds or in-kind contributions. – With Outreach Workgroup, develop a strong program to promote NEMI. – In Phase III expand method matrices beyond water to include solids, air, food, etc. October 11, 2001 (DRAFT 5) MDCB overview 28
Performance Based Systems (PBMS) Workgroup Chair Cliff Annis, J r. , Merck & Co. , Inc. October 11, 2001 (DRAFT 5) MDCB overview 29
Mission of the PBMS Workgroup Lab Performance Field Performance DQOs & MQOs October 11, 2001 (DRAFT 5) Define the dimensions of a performance-based system for field protocols (chemical, microbiological, and biological) and laboratory analyses. Prepare guidance on implementing PBMS in both ambient and compliance monitoring. MDCB overview 30
What is PBMS? A performance based For a performance based system to system permits the use of work, at least 5 darts have to hit the any scientifically appropriate target. . . Well-defined MQOs & DQOs method that demonstrates the ability to meet established performance Adequate supply of reference materials for criteria and complies with method validation specified data quality needs or requirements Known performance characteristics Adequate training in development of MQOs & validation of methods October 11, 2001 (DRAFT 5) Validated or reference methods shown to meet specific MQOs MDCB overview 31
PBMS’s relationship to the framework Lab Performance – The quality of data is related to the performance characteristics of the method used to collect the data. Field Performance DQOs & MQOs – PBMS is directly related to the DQO Process and furthers the collection of appropriate quality data for different monitoring objectives. October 11, 2001 (DRAFT 5) MDCB overview 32
PBMS Workgroup members FEDERAL – – NOAA US DOE USEPA USGS LOCAL/STATE/REGIONAL – – DE River Basin Co East Bay Municipal Utility District CA Department of Toxic Substances Hampton Roads Sanitation District Lab Performance Field Performance DQOs & MQOs INSTITUTES/ORGANIZATIONS – Instant Reference Sources – Standard Methods – ASTM BUSINESS/INDUSTRY – – – Merck & Co. , Inc. American Chemistry Council Hach Co. Catalyst Tetra Tech (contract support) Montgomery Watson October 11, 2001 (DRAFT 5) MDCB overview Broad-based representation and expertise allowed workgroup to reach consensus on PBMS definition & challenges AND allowed group to successfully interact with other national efforts. 33
PBMS Workgroup accomplishments – Reached consensus on conception and definition of PBMS (issue paper being published) – Developed and conducted pilot study addressing certain PBMS implementation issues. Study involved federal, state, municipal, and private labs (submitting study to international journal) October 11, 2001 (DRAFT 5) MDCB overview 34
PBMS Workgroup challenges Pilots and othe Pilots are r activities are needed to conv determine ince organizations t ch hat under whi PBMS is worka conditions ble ork w l l i w S M PB PBMS is a chal lenging and emotional issu e; difficult to effect change o r obtain consensus amo ng agencies October 11, 2001 (DRAFT 5) MDCB overview 35
Where are we going? – Develop further pilots to address PBMS implementation issues – Expand coordination with other organizations in implementing PBMS – Provide usable DQO guidance and promote role of DQOs in water monitoring programs October 11, 2001 (DRAFT 5) MDCB overview 36
How do we get there? – Bring in additional resources and partners to design and implement pilots – Increase dialogue on PBMS between Board and other organizations October 11, 2001 (DRAFT 5) MDCB overview 37
Accreditation Workgroup Co. Chairs Bart Simmo ns, CA Department of Toxic Substance C ontrol Co-chair vac ant October 11, 2001 (DRAFT 5) MDCB overview 38
Mission of the Accreditation Workgroup Lab Performance Field Performance October 11, 2001 (DRAFT 5) Develop and promote a Board position on laboratory accreditation and field certification. Coordinate with external accreditation standardsetting organizations (NELAC). MDCB overview 39
Why Accreditation? There has been the notion that “following the method” ensures accurate data. A method is simply one key component of generating reliable data. Consider an analogy…two chefs, in two kitchens, using the same recipe… of a s d n a h d In the xperience h d, e skille using fres h cook, ts and wit ien d e r g ight in r e h t all nt, a e m p i equ ome c t u o erful d n o w sult. e r l l i w The sa me rec i hands pe in the of an inexpe rience less-th d cook with equipm an-terrific ent is propos a riskier ition. Both cooks, however, may be following the same “method”. October 11, 2001 (DRAFT 5) MDCB overview 40
Accreditation’s relationship to the framework Lab Performance Accreditation… – encourages data sharing and use Field Performance – promotes data consistency – provides an independent objective assessment – promotes uniform quality systems – unifies certification requirements – increases efficiency, potentially lowering costs – promotes flexibility October 11, 2001 (DRAFT 5) MDCB overview 41
Accreditation Workgroup members FEDERAL LOCAL/STATE/REGIONAL – USDOD (Navy) – USGS – USEPA – CA EPA – AZ Department of Health – VA Department of Environmental Quality Lab Performance Field Performance INSTITUTES/ORGANIZATIONS – Standard Methods BUSINESS/INDUSTRY – – American Chemistry Council Catalyst Tetra Tech (contract support) Montgomery Watson October 11, 2001 (DRAFT 5) the strength of the Accreditation Workgroup is its background in ambient water monitoring, which complements the NELAC emphasis on compliance. MDCB overview 42
Accreditation Workgroup accomplishments – Coordination with NELAC and ELAB – White Paper on the value of accreditation – Issue Paper on the need for federal lab accreditation – Diverse representation on the workgroup was key to achieving consensus on recommendations October 11, 2001 (DRAFT 5) MDCB overview 43
Accreditation Workgroup challenges ga n i d n i F on s u s n conse ation it accred ral e of fed ield f d n a s lab n o i t a t i accred Identifyin g key issues to take to NELAC & ELAB Finding effective tactics for achieving objectives October 11, 2001 (DRAFT 5) MDCB overview 44
Where are we going? In 1 year…have clear objectives for field accreditation. . . which can be achieved in about 5 years. October 11, 2001 (DRAFT 5) MDCB overview 45
How do we get there? – Establish a facile communication with NELAC and ELAB – Gain participation in key NELAC committees. October 11, 2001 (DRAFT 5) MDCB overview 46
Water Quality Data Elements (WQDE) Workgroup Co. Chairs Glenn Patte rson, USGS Chuck Job, U SEPA October 11, 2001 (DRAFT 5) MDCB overview 47
Mission of the WQDE Workgroup Data Reporting October 11, 2001 (DRAFT 5) Develop and recommend a core set of data elements for reporting water quality monitoring results, to be voluntarily implemented, that would allow data to be compared regardless of, but recognizing, the purpose of the monitoring activity MDCB overview 48
What is WQDE? – A set of core water quality data elements (metadata) – that will facilitate data sharing between organizations, – allowing the non-collecting data user to assess the comparability of data collected by various entities October 11, 2001 (DRAFT 5) MDCB overview 49
WQDE’s relationship to the framework WQDE is a list of “core metadata, ” facilitating comparability assessments, which tell us: Data Reporting -Who collected analyzed the data -What data were collected -When the data was collected analyzed -Where the data was collected -Why the data was collected -How the data was collected analyzed October 11, 2001 (DRAFT 5) MDCB overview 50
WQDE Workgroup members FEDERAL LOCAL/STATE/REGIONAL – USEPA – USGS – USDOD (Navy) – – – – – AZ Department of Health NJ Geological Survey VA Department of Environmental Quality East Bay Municipal Utility District DE River Basin Commission Hampton Roads Sanitation District FL Department of Environmental Protection NJ Department of Environmental Quality NY Department of Health Washington State Department of Ecology INSTITUTES/ORGANIZATIONS – – – – Association of Public Health Labs National Institute of Standards and Technology CO State University George Washington University Hampton Roads Sanitation District East Bay Municipal Utility Board Standard Methods National Water Research Institute October 11, 2001 (DRAFT 5) Data Reporting NEED SOMETHING HERE BUSINESS/INDUSTRY – Merck & Co. , Inc. MDCB overview 51
WQDE Workgroup accomplishments – Four public meetings were held on the Core Set of Water Quality Data Elements to get broader public input in Chicago, Denver, Washington, D. C. , and the San Francisco Bay area! – Held successful workshop at 2000 national monitoring conference in Austin. – WQDE model was developed and presented to the EPA/State Environmental Data Standards Council for review in March 2001. – Received ACWI endorsement in May 2001! October 11, 2001 (DRAFT 5) MDCB overview 52
WQDE Workgroup challenges e h t g n i Tackl of t n e m p o l e v de ts n e m e l data e logy for bio use d a o r b g Promotin Es D Q W e h of t sed u c o f h g throu ies g e t a r t s outreach October 11, 2001 (DRAFT 5) Conducting pilot studies for t he chemical an d microbiolog ical elements MDCB overview 53
Where are we going? – Developing focused outreach strategies to gain general acceptance of WQDEs – Developing and implementing pilot tests to demonstrate use of WQDEs – Publicizing the WQDEs through state, national, public, private, and professional organizations--promoting voluntary implementation at all levels. – Developing WQDEs for field and biological methods October 11, 2001 (DRAFT 5) MDCB overview 54
How do we get there? – Prepare report to demonstrate implementation successes during pilot testing – Work with various organizations to gain acceptance of WQDEs – Form a Board working group to develop the biology and field methods WQDE list October 11, 2001 (DRAFT 5) MDCB overview 55
Biology Methods Workgroup Co. Chairs Katherine A lben, NYS Department of Health Chris Ingers oll, USGS Donna Fran cy, USGS Mike Miller, WI DNR October 11, 2001 (DRAFT 5) MDCB overview 56
Mission of the Biology Methods Workgroup Data Reporting Lab Performance Field Performance DQOs & MQOs October 11, 2001 (DRAFT 5) Identify, compile, and develop a framework for characterizing and comparing biological methods for water monitoring using: population assessments, molecular biomarkers, and toxicity assays. MDCB overview 57
Why a “Biology Methods” Workgroup? – Wide interest in a framework for comparing biological methods, particularly field population (community methods) – Biological methods create unique challenges in terms of defining method performance, data quality, and method comparability October 11, 2001 (DRAFT 5) MDCB overview 58
Biology Methods relationship to the framework Data Reporting – Coordination and integration with the mission and activities of other workgroups (WQDE, NEMI, PBMS) • Immunoassay, Microbiology, Community/population field methods, Toxicology Lab Performance Field Performance DQOs & MQOs – Target the most important groups of analytes and the most rapidly developing new technologies, in response to information most needed for management of ecological and human health October 11, 2001 (DRAFT 5) MDCB overview 59
Biology Methods Workgroup members FEDERAL – – – USGS USEPA USFWS USDA-NRCS Environment Canada BUSINESS/INDUSTRY – Idexx – Tetra Tech (contract support) LOCAL/STATE/REGIONAL – – – NY Department of Health OH EPA WI DNR East Bay Municipal Utility District Hampton Roads Sanitation District FL DEP MD DNR Los Angeles County Sanitation District DE River Basin Commission Potomac River Interstate Commission Chicago Metropolitan Water Reclamation District – Virginia DEQ – Denver Metro INSTITUTES/ORGANIZATIONS Lab Performance Field Performance DQOs & MQOs Diverse membership biologists and chemists; field and lab management; regulatory and non-regulatory concerns – UT State University – State University of KY – SUNY-Albany School of Health October 11, 2001 (DRAFT 5) Data Reporting MDCB overview 60
Biology Methods Workgroup accomplishments – Attracted wide interest from monitoring community – Workgroup and Board increasingly viewed as coordination venue for a number of organizations – Developed expert Focus Groups for specific types of common biological methods needing attention: Field Population/Community Methods; whole organism toxicity – Bringing in new technologies as a priority to the Methods Board: e. g. , DNA probes, immunoassays, new algal pigment methods October 11, 2001 (DRAFT 5) MDCB overview 61
Biology Methods Workgroup challenges ical Techn es in ng e l l a h c hod t e m g definin mance – perfor rritory; new te ult to diffic in obta nsus e s n o c Developin g summariz format for ing certain types of b methods i iological n. N been a ch EMI has allenging process Diverse a nalytes ( organism whole s; microorg anisms; molecula r biomar kers), and dive rs (water, s e matrices oil, biolo gical) October 11, 2001 (DRAFT 5) MDCB overview 62
Where are we going? – Provide framework/guidance for organizations on how to derive performance characteristics for field population/community and tox methods – Coordinate, communicate, and promote new biological technologies that appear promising for water monitoring (ecological and human health) October 11, 2001 (DRAFT 5) – Coordinate/conduct pilot studies to help define comparability of biological methods, particularly field biological assessment methods MDCB overview 63
How do we get there? – Expanded activities in marine systems – Need more resources – financial and expertise – Especially need more partnerships among EPA, USGS, USFWS, USFS, USDA, NOAA and states October 11, 2001 (DRAFT 5) MDCB overview 64
Nutrient Methods Workgroup Chairs Ed Santoro, DE River Basin Comm ission Ron Jones, FL Internation al University October 11, 2001 (DRAFT 5) MDCB overview 65
Mission of the Nutrients Workgroup Data Reporting Lab Performance Field Performance DQOs & MQOs October 11, 2001 (DRAFT 5) Develop information to respond to the needs of the PBMS, NEMI, and WDQE Workgroups and respond to other nutrient methods issues MDCB overview 66
Why a “Nutrients” Workgroup – Nutrient criteria and environmental issues regarding nutrients have high visibility. Field sampling and lab analytical method issues are a factor. – Nutrients are a major non-point TMDL pollutant – The quality of nutrient methods being used is widely divergent. October 11, 2001 (DRAFT 5) MDCB overview 67
Nutrient’s relationship to the framework Data Reporting – Nutrients in public eye and have special methodological issues in terms of data quality and comparability – Workgroup is providing important input to both PBMS and NEMI Lab Performance Field Performance DQOs & MQOs – Field sampling issues are a critical aspect of nutrient data October 11, 2001 (DRAFT 5) MDCB overview 68
Nutrients Workgroup members FEDERAL LOCAL/STATE/REGIONAL – USEPA – USGS – – – DE River Basin Commission NYS Health Department KY Department of Water Hampton Roads Sanitation District WI DNR VA DEQ Data Reporting Lab Performance Field Performance DQOs & MQOs INSTITUTES/ORGANIZATIONS – – ASTM FL International University Kent State University SUNY-Albany School of Healthy cross-section of members from federal, state, and academia…and they put up with Ed. . . BUSINESS/INDUSTRY – Tetra Tech, Inc (contract support) – Merck & Co. , Inc. October 11, 2001 (DRAFT 5) MDCB overview 69
Nutrients Workgroup accomplishments – Broad-based representation including state, private, and federal organizations – Developed White Paper regarding EPA draft nutrient criteria based on consensus of many organizations; submitted to EPA – Produced technical report on nutrient comparability and compilation of performance information for different nutrient methods October 11, 2001 (DRAFT 5) MDCB overview 70
Nutrients Workgroup challenges ne i r a u t s /e Marine ave not h issues essed r d d a been k of c a l f o e becaus rtise expe October 11, 2001 (DRAFT 5) Need mor e participat active ion from regulatory agencies ( EPA, state e. g. , s) other reso and u agencies ( rce e. g. , NOAA) MDCB overview 71
Where are we going? – Coordinate with EPA in providing appropriate DQOs and methods for nutrient compliance monitoring – Perform and publish pilot studies addressing PBMS issues using nutrient field and lab methods – Coordinate guidance/ recommendations on appropriate field and lab methods for nutrient analysis given DQOs – Develop final draft of nutrient pilot paper October 11, 2001 (DRAFT 5) MDCB overview 72
How do we get there? – Resources – financial as well as expertise – Need more participation from federal agencies: NOAA, USDA, USFS, others October 11, 2001 (DRAFT 5) MDCB overview 73
- Slides: 73