Preparing Quality Assurance Project Plans Presented By Denise

Preparing Quality Assurance Project Plans Presented By: Denise L. Goddard, Chemist Quality Assurance Section Athens, Georgia

EPA DISCLAIMER n This Presentation is for Training Purposes Only.

EPA - QA Documents for Preparing Quality Assurance Project Plans n n n Guidance on Systematic Planning Using the Data Quality Objectives Process, EPA QA/G-4, EPA/240/B-06/001 (February 2006) Requirements for Quality Assurance Project Plans, EPA QA/R-5, EPA/240/B 01/003 (March 2001) Guidance for Quality Assurance Project Plans, EPA QA/G-5, EPA/240/R-02/009 (December 2002)

Are QAPPs Really Required? ? ? YES!! n Quality System Requirements – “Approved Quality Assurance Project Plans (QAPPs) or equivalent documents defined by your organization’s QMP, for all applicable projects and/or studies that will involve environmental data collection or where environmental decisions will be made for a particular site. QAPP must be approved prior to any data gathering work or activities, except under circumstances requiring immediate action (emergency response) to protect human health and the environment or operations conducted under police powers. ”

Organizational Applicability? ? n EPA Organizations – Covered under Executive Order 5360. 1, A 2: “The Agencywide Quality System requirements defined by this Order apply to all EPA organizations, and components thereof, in which the environmental programs conducted involve the scope of activities described in Section 5. a above. The authority of this Order applies only to EPA organizations except as addressed by Section 5. d(2) below. ”

External Organizations Requirements n Extramural Agreements: Agency-wide Quality System requirements may also apply to non-EPA organizations. These requirements are defined in the applicable regulations governing extramural agreements. Agencywide Quality System requirements may also be invoked as part of negotiated agreements such as memoranda of understanding (MOUs). Non -EPA organizations that may be subject to quality system requirements include:

Extramural Agreements q q q (a) Any organization or individual under direct contract to EPA to furnish services or items or perform work (i. e. contractor) under the authority of 48 CFR 46, (including applicable work assignments, delivery orders, and task orders); 40 CFR 31 – Grants & Cooperative Agreements with State & Local Governments 40 CFR 35 – State & Local Assistance

The Purpose of a Quality Assurance Project Plan n As a planning document, the QAPP should contain a detailed description of environmental data collection activities and operations, the problems associated with a site, the sampling and analysis requirements, the decisions to be made, and the necessary QA/QC activities governing this effort.

Issues Addressed by a QAPP n n The QAPP must provide sufficient detail such as: The project’s technical and quality objectives – these must be well defined and agreed upon by all affected parties and stakeholders The program-specific and site-specific requirements (stipulated in consent decrees, records of decision, regulations, statutes, etc. ). The intended measurements, data generation or data acquisition methods that are appropriate for achieving project goals/objectives.

Issues Addressed by a QAPP – Con’t n n A summary of the assessment procedures for confirming that data of the type, quantity and quality required and expected were obtained, and A description of the process for evaluating the limitations on the use of the information or data obtained that includes identifying, documenting and communicating the limitations to all affected parties and stakeholders.

Overview of Content Requirements n To be effective, the QAPP must clearly state: q q q The purpose of the environmental data operation (e. g. , enforcement, research and development, rulemaking), The type of work to be done (e. g. , pollutant monitoring, site characterization, risk characterization, bench level proof of concept experiments), and The intended use of the results (e. g. , compliance determination, selection of remedial technology, site closure, development of environmental regulations).

n Before We Start - Some Preliminaries – Format/Content Because the Requirements QAPP is a formal document – it should contain: q q A Title Page containing the title of the document, the Identification of the Organization that Prepared the QAPP, the Preparation Date and the Version Number – The document should be Paginated An Approval Page – Containing Signature and Date Blocks for each of the individuals/organizations responsible for approving this document.

Some Preliminaries – Format and Content Requirements q q A Distribution List – Containing the Names, Mailing Addresses, Phone Numbers, and Email Addresses for each of the individuals and organizations requiring copies of the approved QAPP. Table of Contents – For Text, Tables, Figures, Maps & Appendices. If there are numerous Tables, Figures & Maps – Place these items in the Appendix to reduce breakup of the text.

Some Cautionary Tips!!! n Some Cautions: q q q Avoid using generic language that does not provide the required information or level of detail required. For projects requiring the generation of chemical or biological data, make sure that you produce a list of contaminants of concern – or identify the biological parameters of interest. Make sure the approved QAPP is distributed to project personnel, laboratory staff and if you are using CLP, identify the COCs in project log (unless there are numerous contaminants).

Let’s Start - Components of a QAPP n n n A QAPP is composed of approximately 25 elements that are grouped into four classes or categories as follows: Class A – Project Management Class B – Measurement/Data Acquisition Class C – Assessment/Oversight Class D – Data Validation/Data Usability

Class A Topics - Overview n The elements in this group address: q q Project Management Project History/Site History Goals & Objectives of the Project Outputs

Class A Topics n The following topics must be addressed as part of the Class A components/elements: q q q q A 1 – Title/Approval Page A 2 – Table of Contents A 3 – Distribution List A 4 – Project/Task Description A 5 – Problem Definition/Background Info A 6 – Project/Task Description A 7 – Quality Objectives & Criteria – DQOs/DQIs A 8 - Documents & Records

A 4 Project/Task Organization n The following information is required: q q q Identify the individuals/organizations that will participate in the project/study – discuss their roles/responsibilities – identify the principal data users, decision makers, QA Manager, stakeholders and end data users. QA Manager – should be iidentified in the QAPP – this individual should be independent of data collection operations, should have direct access to senior management, have overall authority over data collection activities when non-conformance with the QAPP is encountered. An organizational chart depicting the lines of communication and authorities between senior management, the QAM and project personnel – should also include principal and end data users, decision makers, stakeholders, contractors & and any subcontractors.

Organizational Chart 1 Senior Management Field Sampling Staff Laboratory Analysis Data Validation

Organizational Chart 2 Senior Management Field Sampling Staff Laboratory Analysis QA Manager Data Validation Data Quality Assessment

Organizational Chart 3 Senior Management QA Manager Field Sampling Staff Laboratory Analysis Data Validation Data Quality Assessment

Organizational Chart 4 Senior Management QA Manager Project Management Field Sampling Staff Laboratory Analysis Data Validation & Data Quality Assessment

Organizational Chart 5 Senior Management Project Manager Field Sampling Staff Joe Smo Jane Doe QA Manager Decision Makers Laboratory Analysis Organic Analysis Inorganic Analysis Data Validation & Data Quality Assessment John WU Linda Good DQA D. Val.

A 5 Problem Definition/Background n n n Summarize the problem to be solved The decision to be made Or outcome to be achieved Include background/historical information Include scientific and regulatory perspectives

A 6 Project/Task Description n n Summarize all work to be done Specify all measurements that must be taken – identify which measurements are critical or noncritical - critical measurements will be used to make site decisions – non-critical measurements won’t be used during the decision making process Provide a list of all of the equipment required Identify any products that will be produced Provide Maps, Charts, Figures & Tables

A 7 Quality Objectives & Criteria n n Describe the quality goals/objectives for the project – provide the performance criteria for achieving these goals/objectives, etc. Provide the project-specific data quality objectives (both qualitative and quantitative) and the specific data quality indicators (precision, bias, sensitivity, comparability, completeness and representativeness) relevant to the project/study.

Brief Overview of the Systematic Planning Process n Data Quality Objectives Process: q q q q Step 1 – State the Problem Step 2 – Identify the Goals of the Study Step 3 – Identify the information inputs Step 4 – Define the Boundaries of the Study Step 5 – Develop the Analytical Approach Step 6 – Specify Performance or Acceptance Criteria Step 7 – Develop the Plan for Obtaining Data

Additional Thoughts on the DQO Process n n Include any and all assumptions concerning site contamination, contaminant pathways, remedial techniques, clean-up design, monitoring strategies, etc. , as part of the DQO process. Identify any suspected potential departures from assumptions in support of the DQO process.

A 8 Special Training/Certifications n n Identify and describe any specialized training (including QA training) needed by project personnel required to successfully complete the project or task. Discuss how such training will be provided – discuss who is responsible for obtaining internal training for staff. Discuss where training documentation will be maintained. Specify whether professional certifications, accreditations or licenses are required for staff to perform their designated tasks/duties.

A 9 Documents & Records n n Describe the process and responsibilities for ensuring the appropriate project personnel have the most current approved version of the QAPP, including version control, updates, distribution and disposition. Itemize the information required in project documents, records and reports. The type of information required for analytical data reports must be specified for both hard-copy and electronic formats. Data deliverables can and do include raw data, data from other sources such as computer databases, literature searches, field logs, sample preparation logs, analysis logs, instrument printouts, model inputs and outputs files, and the results of calibrations and QA checks.

A 9 Documents & Records n n n Specify whether status/progress reports and final reports are required. Specify or reference all applicable requirements for the final disposition of records/documents, including location and retention time. Identify the individuals who are responsible for preparing project documents, records and reports – also identify who within EPA will receive this information.

Class B Topics - Overview n n n Discuss all aspects of data collection and generation Describe sampling design and provide rationale for your approach Specify the analytical measurements both field and fixed laboratory Describe sample handling and chain-of-custody requirements Specify QA/QC samples with acceptance criteria

Class B Topics n n n n n B 1 – Sampling Process Design B 2 – Sampling Methods B 3 – Sample Handling & Custody B 4 – Analytical Methods B 5 – Quality Control B 6 – Instrument/Equipment Testing, Inspection & Maintenance B 7 – Instrument/Equipment Calibration & Frequency B 8 - Inspection/Acceptance of Supplies & Consumables B 9 – Non-Direct Measurements B 10 – Data Management

B 1 Sampling Process Design & Experimental Design n Describe the experimental data generation or data collection design for the project, including as appropriate: q q q The types & numbers of samples required The design of the sampling network The sampling locations, frequency of collection at each location and sample matrices The measurement parameters of interest, and The rationale for the sampling design chosen.

Sampling Designs Should be Consistent with your Conceptual Models!!assumptions n Evaluate your underlying n n whether they are conscious or unconscious Use a statistical tool or sampling tool such as Visual Sample Plan to test your sampling design. Use historical data if available to determine the actual distribution of contaminants.

B 1 Sampling Designs n Directed Sampling Designs q n Judgmental Sampling Probability Sampling Designs q q q q Simple Random Systematic/Grid Stratified Composite Adaptive Collaborative ( Double) Hot Spot

Judgmental Sampling Design Pros n q Judgmental sampling is the subjective selection of sampling locations in space & time by an individual analyst or expert. Consistent with intuitive feeling q q q Easy to direct, easy to do May be cost effective if the conceptual site model for the project is correct Great if you know absolutely everything there is to know about the site and your conceptual site model is absolutely correct.

Judgmental Sampling Design Cons n Inference from sample to population questionable q q q Use of incorrect conceptual model can lead to incorrect decisions – can be a disaster. Not suitable for estimating underlying population parameters (e. g. , mean) with specified confidence – Cannot use statistics to evaluate distribution of data with any degree of confidence – with this sampling design this is no underlying assumption that the data are normally distributed. Not suitable for testing hypothesis about underlying populations with specified decision error rates

Simple Random Sampling Pros n Simple in concept and provides proper (theoretical support) data for statistical data analysis – representative sampling locations are chosen using theory of random chance probabilities q q Protects against bias in estimating parameters (e. g. , means) and testing hypothesis Is the basic building block of more complicated (and efficient) sampling designs.

Simple Random Sampling Cons n Ignores available information that could be used to develop more cost-effective sampling designs q q q Not as effective as other designs for delineating patterns of contamination or finding hot spots Difficult to find randomly selected sampling locations Tends to demand large numbers of samples

Systematic (Grid) Sampling n Systematic (grid) sampling consists of collecting samples according to a specified pattern at regular intervals in space or time within a grid pattern: q q Square or rectangular grid patterns over space Equal-interval sampling along a straight line

Systematic Sampling - Pros n n Easy to explain and implement and provides uniform coverage of site or project Good for estimating boundaries, trends, or patterns of contamination over space or time. May yield more precise estimates of population parameters than other sampling designs Required for statistical data analysis to estimate trends and spatial patterns

Systematic Sampling - Cons n n Systematic sampling can cause estimated means to be biased if the sampling grid pattern lines up with any pattern of contamination. More information is needed (than for simple random sampling) about the population to estimate the variance of the estimated mean.

Stratified Sampling n n The target population is divided meaningfully into contiguous sub-populations called strata Sampling locations are selected independently within each strata using some sampling design

Stratified Sampling - Pros n n Dramatically reduces the variability present in the population and hence improves precision Enables estimates of individual areas to be made Assists in providing good coverage of the project Allows for increased samples from policy or project sensitive areas

Stratified Sampling - Cons n n n Requires advanced knowledge in order to divide the study area into roughly homogeneous strata before sampling The number of samples to be taken in each stratum must be determined If strata boundaries are inaccurate, what appears to be outlier data can appear due to being in the wrong strata

Composite Sampling n n n Many individual (grab) samples are combined and thoroughly mixed to make a homogeneous whole. At random, sub-samples (composite samples) are made and sent to the laboratory for analysis. The physical size of composite samples are the same size as those obtained at random.

Composite Sampling - Pros n n Allows for estimating the mean concentration with the same precision at a lower cost Provides better coverage of the study site without increasing the number of chemical analyses Allows for a more representative sample from a basic area of sample support (sampling unit). Can be used in combination with other sampling designs.

Composite Sampling - Cons n n Information on individual samples used to form composite samples is lost in compositing Potential for loss of contaminants (volatiles) during the mixing and handling phase Potential for reactions and interactions among analytes during compositing Need to make decision on how many grab samples to be composited and how many composite samples to send for analysis

WHY IS YOUR SAMPLING DESIGN IMPORTANT!! n n n UNCERTAINTY!!! Due to the Variability Between Analytical Results Within a Given Data Set? ? ? OR Due to Sampling Issues? ? ?

And The Correct Answer Is n BOTH!!! n ANYTHING ELSE? ? ? You bet, in addition to how you collected the samples is the important issue of WHERE you collected your samples and this relates back to your sampling design and the assumptions you made concerning site conditions which in turn directed the development of your conceptual site model – these issues could have greatly increased your uncertainty and may lead to a wrong decision. A wrong sampling design and a flawed conceptual site model will lead to DECISION ERROR.

Bottom Line!! n ‘It is understandable that analytical studies, with their sophisticated instrumentation and high cost, are often perceived as the dominant element in a site characterization project/study. Yet, despite that sophistication and high cost, analytical data generated under a scientifically defective or unsound sampling design will have limited utility. ’

The Best Result n Data Set Distribution – Normality

Normal Distributions & the Central Limit Theorem n The normal distribution is one which appears in a variety of statistical applications. One reason for this is the central limit theorem. This theorem tells us that sums of random variables are approximately normally distributed if the number of observations is large. For example, if we toss a coin, the total number of heads approaches normality if we toss the coin a lot of times. Even when a distribution may not be exactly normal, it may still be convenient to assume that a normal distribution is a good approximation. In this case, many statistical procedures, such as the t-test can still be used.

Ranked Set Sampling – A Combination of Statistics & Expert Judgment n n A sampling design where expert judgment is used in combination with simple random sampling Simple random sampling is used to create a large number of potential samples. The expert then ranks these potential samples and selects which to send for analysis.

Ranked Set Sampling – Pros & Cons n Pros: q q q n Better representativeness through using experts Better precision than random sampling Same simple formulae to use Cons: q q Increased cost of the expert ranking samples Difficult quantifying exact improvement Need to find best variable to do the ranking on …. . but the pros definitely outweigh the cons

B 2 Sampling Methods n n n n Describe the procedures for collecting samples – provide SOPs Specify sampling methods and equipment Provide sample container, volume, preservation, and holding time requirements Describe the decontamination procedures Provide a list of sampling equipment Describe performance requirements for sampling methods Identify the location of support facilities Identify the individuals who are responsible for implementing corrective actions during field sampling activities

B 3 Sample Handling & Custody n n n Describe the requirements for sample handling & custody in the field, laboratory, and during transport, taking into your holding time requirements. Include sample handling requirements for packaging, transporting and storing the collected samples. Provide examples of sample labels, custody forms, sample custody logs and custody seal.

B 4 Analytical Methods n n n n Identify the analytical methods, instruments, equipment required. Discuss how laboratory staff are to sub-sample the collected environmental sample. Identify the contaminants of concern and specify the extraction, digestion and analytical method for each contaminant Specify the laboratory decontamination and waste disposal procedures Identify the individuals who are responsible for implementing corrective actions when problems are encountered during extraction, digestion or analysis of the samples. Specify the detection limit requirements for each contaminant. Provide the regulatory standard(s) (action limits, ARARs, MCLs, water quality standards, etc. ).

B 5 Quality Control n Identify QC activities needed for each sampling, analysis, or measurement technique. For each required QC activity, list the associated method or procedure, acceptance criteria, and corrective action.

B 5 Quality Control Samples n Specify the type and frequency of quality control sample collection or QC activity: q q q Blanks Spikes (MS/MSDs) Duplicates Standard Reference Materials Rinsates/Equipment Blanks Second Column Confirmation

B 5 Quality Control Samples n n Specify the acceptance criteria for spike recoveries and the precision requirements. Specify the frequency of QC sample collection and analysis.

B 6 Testing, Inspection & Maintenance n n n Identify the instruments/equipment requiring testing, inspection & maintenance during data collection operations (both field and fixed laboratory). Provide the testing, inspection and maintenance procedures & identify the individuals who are responsible for these tasks. Specify the frequency of instrument & equipment testing, inspection & maintenance. Discuss the corrective actions necessary when instruments & equipment no longer function as required. Identify the location of spare parts for repairing items.

B 7 Calibration & Frequency n Identify all tools, gauges, instruments and other sampling, measuring and test equipment used for data generation or collection activities affecting the quality that must be controlled and, at specific periods, calibrated to maintain performance within specified limits.

B 7 Calibration & Frequency n n n Identify the instruments/equipment requiring calibration. Describe the calibration procedures and identify the standards used during calibration. Specify the frequency of calibration and specify the acceptance criteria for calibrations (for all instruments/equipment). Identify the individuals who are responsible for calibrating instruments/equipment. Identify the individuals who are responsible for performing calibrations.

B 8 Supplies & Consumables n n Identify the supplies & consumables that are used during field data collection operations Supplies & consumables would include calibration solutions/standards, calibration gases, reagents, tubing & hoses, de-ionized water, potable water, electronic storage media (data loggers), etc. Specify the acceptance and rejection criteria for each item. Identify the individuals who will inspect supplies & consumables to ensure that they meet the relevant acceptance criteria.

B 9 Non-Direct Measurements n n Identify any types of data needed for project implementation or decision making that are obtained from non-measurement sources such as computer data bases, programs, literature searches, surveying data and historical data/data-bases, modeling, etc. Describe the intended use of this data, and define the acceptance criteria for the use of such data in the project and specify any limitations and restrictions in the use of the data.

B 10 Data Management n n n Describe the project data management process, tracing the path of the data from their generation to their final use or storage (e. g. , the field, the office and/or the laboratory). Describe or reference the standard record-keeping procedures, document control system, and the approach used for data storage and retrieval on electronic media. Discuss the control mechanism for detecting and correcting errors and for preventing loss of data during data reduction, data reporting, and data entry to forms, reports and databases. Provide examples of any forms or checklists to be used.

B 10 Data Management n Identify and describe all data handling equipment and procedures used to: q q q q Process Compile And analyze data Including computer hardware Computer software Software configurations Include secondary data sources

B 10 Data Management n n Describe the procedures that will be followed to demonstrate acceptability of the hardware and software configuration required, and describe the process for assuring that applicable information resource management requirements are satisfied. Discuss how your organization will comply with EPA data management requirements as specified in EPA Order 2180. 1 or newly issued data standards.

Class C Topics - Overview n The topics in this group address the activities for assessing the effectiveness of project implementation and associated QA/QC activities. The purpose of assessment is to ensure that the QAPP is implemented as prescribed.

Class C Topics n n C 1 – Assessment & Response Actions C 2 – Reports to Management

C 1 Assessments & Response Actions n n n Describe each assessment to be used in the project including the frequency and type. Assessments include, but are not limited to, surveillance, management systems reviews, readiness reviews, technical systems audits, performance evaluations, audits of data quality and data quality assessments. Discuss the information expected and the success criteria (i. e. , goals, performance objectives, acceptance criteria specifications, etc. ).

C 1 Assessments & Response Actions n n n List the approximate schedule of assessment activities. For any planned self assessments (utilizing personnel from within the project groups) identify potential participants and their exact relationship within the project organization. For independent assessments, identify the organization and the person(s) that shall perform the assessments if this information is available. Describe how and to whom the results of each assessment shall be reported. Discuss how corrective actions will be implemented, documented, tracked and verified for closure.

C 2 Reports to Management n n Identify the frequency and distribution of reports issued to inform management (EPA or otherwise) of the project status, or to inform them of the results of performance evaluations and systems audits, data quality assessments, and significant data quality issues. Identify the preparer and the recipients of the reports and any specific actions recipients are expected to take as a result of the reports.

Class D Topics - Overview n The topics in this group address the QA activities that occur after the data collection phase of the project is completed. Implementation of these elements determines whether or not the data conform to the specified criteria, thus satisfying the project objectives.

Class D Topics n n n D 1 – Data Review, Verification & Validation D 2 – Verification & Validation Methods D 3 – Reconciliation with User Requirements

D 1 Data Review, Verification & Validation n n Specify the criteria used to review and validate the data – that is provide the acceptance and rejection criteria by which the data will be assessed to determine the quality of this information. Provide a list of the data qualifier flags or qualifiers along with their respective definitions.

D 2 Verification & Validation Methods n n n Describe the process to be used for verifying and validating the data, including the chain-of-custody for data throughout the life of the project or task. Discuss how issues shall be resolved and the authorities for resolving such issues within the organization. Describe how the results of data verification & validation are conveyed to end data users, decision makers and stakeholder. Precisely define and interpret how validation issues differ from verification issues for this project. Provide examples of any forms or checklists to be used; and, identify any project-specific calculation required.

D 2 Reconciliation with User Requirements n n n Describe how the results obtained from the project or task will be reconciled with the requirements defined by the data user or decision maker. Outline the proposed methods to evaluate the data and determine those possible anomalies or departures from assumptions that were established in the planning phase of data collection. Describe how reconciliation with user requirements will be documented, issues will be resolved, and how limitations on the use of the data will be documented, communicated and reported to decision makers and stakeholders.

Reference Page & Appendices n n Reference Page: Contains a list of the references cited in the QAPP. Appendices: Contains any relevant materials and documents that will support the QAPP.

QAS Contacts n Marilyn Maycock, Chief q q n (706) 355 -8553 maycock. marilyn@epa. gov Denise Goddard, Chemist q q (706) 355 -8568 goddard. denise@epa. gov

QAS Contacts n Charlie Appleby, Chemist q q n (706) 355 -8555 appleby. charlie@epa. gov Ray Terhune, Chemist q q (706) 355 -8557 Terhune. ray@epa. gov