Headquarters U S Air Force Integrity Service Excellence

Headquarters U. S. Air Force Integrity - Service - Excellence SOURCE SELECTION TRAINING (EXECUTION) <acquisition title> <Trainer’s name> <Trainer’s organization> Be America’s Best…War-Winning Capabilities on Time, on Cost 1

Introductions Facilities Orientation Classroom Etiquette Ø Start/Stop Times Ø Blackberry’s/PDA’s – Please Turn OFF! Class interaction is encouraged - - this is not a lecture! Introductions Be America’s Best…War-Winning Capabilities on Time, on Cost 2

Overview Learning Objectives RECAP – What Team Has Accomplished Basic Purpose of SS Source Selection Process Roadmap & AF SS Process Schedule Source Selection Organization Ø Comp Range Elimination, Discussions, Request for Final Proposal Revision Initial Evaluation Phase Ø Proposal Receipt, SS Documentation & File Mgm’t, EZ Source Tool, Basic Evaluator Guidelines Initial Evaluation Activities Ø Technical, Past Performance, Cost/Price, ENs, Comp Range Final Evaluation Activities Ø Proposal Evaluation, SSEB PAR, SSAC Comparative Analysis Report, Decision Brief Ø SSO Roles & Responsibilities Pre-FPR Evaluation Activities Award and Debriefing Activities Records Retention Ethics and Procurement Integrity What is Source Selection Information? References Summary Be America’s Best…War-Winning Capabilities on Time, on Cost 3

Example Workshop Schedule – Day 1 0800 – 1130 Introductions/ Overview/Schedule Learning Objectives RECAP Source Selection Organization Source Selection Process Initial Evaluation Activities 1130 – 1230 LUNCH 1230 – 1600 Initial Evaluation Activities (cont. ) Pre-Final Proposal Revision Evaluation Activities Final Evaluation Activities Breaks – As Required Be America’s Best…War-Winning Capabilities on Time, on Cost 4

Example Workshop Schedule – Day 2 0800 – 1130 Recap from Day 1 Ethics and Procurement Activities Summary Award Contract and Debriefing Activities Mock Source Selection Activity (Optional) 1130 – 1230 Lunch 1230 - 1600 Breaks – As Required Be America’s Best…War-Winning Capabilities on Time, on Cost 5

Learning Objectives After completion of this training AF Source Selection Teams will understand: Ø Roles and responsibilities Ø Processes/procedures to evaluate proposals Ø Methodology and requirements for creation and maintenance of documentation Ø Approaches and tools available Ø Necessity of conducting thorough and timely debriefings Ø Importance and applicability of ethics and procurement integrity rules throughout the process Be America’s Best…War-Winning Capabilities on Time, on Cost 6

Basic Purpose of Source Selection To select the contractor (the source) Ø Whose proposal meets our requirements and provides the best value ü “Best Value” means expected outcome of an acquisition that provides greatest overall benefit in response to requirement Ø That is most likely to perform all requirements Ø That is best able to handle problems that might occur during contract performance Ø That provides most realistic plan for contract performance Be America’s Best…War-Winning Capabilities on Time, on Cost 7

Recap – What Team Has Accomplished Issued final Solicitation/RFP incorporating Ø Requirements Documents Ø Source Selection approach ü Evaluation Methodology Ø Key aspects of approved Source Selection Plan Ø RFP is compliant with approved Acquisition Plan Established Source Selection Team Ø All members have reviewed Source Selection Plan and RFP Ø All members and advisors have signed Non-Disclosure Agreements at start of SS and COIs after receipt of offerors’ proposals A successful Source Selection requires a team that is engaged and prepared Be America’s Best…War-Winning Capabilities on Time, on Cost 8

Air Force Source Selection Process – Proposal Evaluation Be America’s Best…War-Winning Capabilities on Time, on Cost

Schedule <Trainer to insert the Source Selection Milestone Schedule Be America’s Best…War-Winning Capabilities on Time, on Cost 11

Source Selection Organization <Trainer to insert the Source Selection Organization Be America’s Best…War-Winning Capabilities on Time, on Cost 12

SSA - Role and Responsibilities Responsible for Proper and Efficient Conduct of Source Selection Process Your SSA: - Approved Source Selection Plan - Appointed chair for SSEB and SSAC (if used) - Established realistic schedule - Ensured all SST members are properly trained and experienced - Ensured execution of Non-Disclosure agreements and COI statements - Determine whether to award with or without discussions - Select the Best Value source - Document the decision Be America’s Best…War-Winning Capabilities on Time, on Cost 13

PCO - Roles and Responsibilities Primary Business Advisor and Principal Guidance Source Throughout Source Selection PCO Shall: - Manage all business aspects of acquisition and advise and assist SSA in execution of responsibilities - Work with SSEB Chair to ensure evaluation is consistent with RFP - Establish safeguards and procedures to protect SS information - Obtain required approvals for Non-government advisors before they start any support activities - Maintain Source Selection documents and official file - Release RFP - Serve as single point of contact for all solicitation-related inquiries - Control exchanges with offerors, after receipt of proposals - Establish competitive range, after SSA approval to enter into discussions - Complete SS participants worksheet (ACAT I/II) after award Be America’s Best…War-Winning Capabilities on Time, on Cost 14

SSAC - Roles and Responsibilities SSAC Established by SSA to Gain Functional Expertise and Support Your SSAC: - Oversee SSEB - Review SSEB evaluation results to ensure that it follows evaluation criteria and that the ratings are appropriate and consistent - Provide written comparative analysis and recommendation to SSA - Ensure minority opinions within the SSAC are documented Be America’s Best…War-Winning Capabilities on Time, on Cost 15

SSEB Chairperson – Roles and Responsibilities SSEB Chair Manages Evaluation Team and Acts as Interface to SSA and SSAC SSEB Chair shall: - Appoint SSEB members - Establish functional evaluation teams - Ensure SSEB members know HOW to evaluate proposals BEFORE they review proposals - Staff SS team with proper skill and experience members - Allow adequate time to successfully complete the SS - Ensure adequate resources are available - Provide consolidated results - Document SSEB evaluation results - Support any post SS activities Be America’s Best…War-Winning Capabilities on Time, on Cost 16

SSEB Evaluators – Roles and Responsibilities SSEB Evaluators are Generally Organized Into Functional Areas SSEB Evaluators shall: - Conduct comprehensive review and evaluation of proposals against solicitation requirements and approved evaluation criteria - Assist SSEB Chair in documenting evaluation results - Support any post-award activities Be America’s Best…War-Winning Capabilities on Time, on Cost 17

Advisors – Roles and Responsibilities Two Distinct Types Of Advisors: Government And Non-government Government advisors shall: - Assist SSA - if no SSAC used - Assist SSEB as subject matter experts Non-Government advisors shall (Includes FFRDCs): - Be used on a limited basis - Be approved via written determination (FAR 37. 203/37. 204) - Sign a Non-Disclosure Agreement (NDA), Conflict of Interest Statement, and provide documentation regarding personal stocks owned - Not rate or rank any offeror’s proposal - Not participate in any past performance evaluation or access offerors’ PPI Be America’s Best…War-Winning Capabilities on Time, on Cost 18

Initial Evaluation Activities Proposal Receipt PCO ensure: Proposal Validity Page Limit Compliance Correct number of copies (Paper & Electronic) Master Table of Contents is correct All erroneous and extraneous material is identified Be America’s Best…War-Winning Capabilities on Time, on Cost 19

Proposal Receipt (cont. ) What will you use to capture evaluation and subsequent documentation? What Source Selection software (EZ Source or other) is your team using? Ø Has all proposal data been loaded? Ø Have team members been adequately trained? Have past performance volumes been received? Ø Will past performance team use past performance questionnaires? <Trainer to update past performance information based on RFP> Be America’s Best…War-Winning Capabilities on Time, on Cost 20

SS Documentation & File Management Emphasize standardization for configuration control Establish ROEs for data management & accountability Ø Increases ability to audit & establish completeness of record Fully understand capabilities & limitations of SS tools Ø SS tool does not contain entire record Appoint SS Records Custodian Ø Recommended best practice Document management, file structure, and disposition Source Selection Documentation Guidance Be America’s Best…War-Winning Capabilities on Time, on Cost 21

EZ Source Tool Mandatory Use Criteria for Competitive Acquisitions Ø Value $50 M or greater Ø Hi-Vis Programs ü Consolidation of individual contracts ü Re-competition as a result of sustained protest (corrective action) ü Political sensitivity ü PEO or CC deems high-risk program regardless of $$ value or ACAT level Deployment of EZ Source will be accomplished in two spirals over next 18 months (beginning Jun 2013) Ø Spiral I locations meeting mandatory criteria must use EZ Source if solicitation is issued on or after location’s effective date (Policy Memo 13 -C-02) Ø Spiral II - all remaining locations (estimated Oct 2014? ? ) Be America’s Best…War-Winning Capabilities on Time, on Cost 22

Source Selection Tools POCs EZ Source tool (AFMC) Ø POC: Kerry Estes, DSN 785 -5471, kerry. estes@wpafb. af. mil EZ Source Templates Electronic Source Selection (ESS) tool (SMC) Ø POC: Mr. Ted Nguyen, DSN 633 -5072, ted. nguyen@losangeles. af. mil Decision Point Ø See www. acqcenter. com for more information Fed Select Ø See www. caci. com Ø POC: Steve Ford, (703) 486 -3266 ext 1031, sford@caci. com Be America’s Best…War-Winning Capabilities on Time, on Cost 23

Be America’s Best…War-Winning Capabilities on Time, on Cost

Initial Evaluation Phase Each Evaluation Team assesses proposals for its factor – Technical, Past Performance, Cost/Price Ø Cost/Price Factor is evaluated IAW Section M, but not rated Individual evaluators generate facts and findings in their assigned areas and record them on worksheets Ø Team Leaders combine information from individual Worksheets into Summary Sheets and then rate their subfactors and factors Ø Team Leaders generate their portion of the SSEB Proposal Analysis Report (PAR) using Summary Sheets as a basis ü SSEB PAR and Summary Sheets are used to generate SSAC/SSA briefing slides Be America’s Best…War-Winning Capabilities on Time, on Cost 25

Flow of Evaluation Findings SSAC BRIEF Be America’s Best…War-Winning Capabilities on Time, on Cost 26

Basic Evaluator Guidelines Perform your own independent assessments, use your knowledge and experience Thorough narratives are essential to producing a fair, consistent, fact-based and discriminating evaluation Ø Basis for SSAC/SSA trade-off analysis Ø Narratives tied to proposal and RFP requirements Ø Assessments clearly describe benefits and/or detriments Coordinate & communicate among teams through Team Leaders Ø Assure integrated evaluation of overlapping areas Ø Inform others of info that may help evaluation of their areas Ø Talk with fellow evaluators and Team Leader daily; however don’t let anyone unduly influence your evaluation Be America’s Best…War-Winning Capabilities on Time, on Cost 27

Initial Evaluation Activities Team Leads: Ø Have all evaluators been trained? ü Are evaluators ready to begin this task? Ø Have you designated specific forms or worksheets? Ø How will you evaluate offeror’s proposals? ü All evaluators review one proposal before moving to next OR ü Each evaluator reviews only one proposal OR ü Each evaluator reviews by subject matter OR ü Different approach? Ø How will you decide an evaluator’s write up is sufficient? Be America’s Best…War-Winning Capabilities on Time, on Cost 28

Initial Evaluation Activities (cont. ) Team Leads: does your approach. . . Ø Ensure integrated assessment of proposal? Ø Facilitate evaluation against published criteria? Ø Prevent comparison of proposals? Ø Encourage efficiency and consistency? Ø Prevent transfer of one offeror’s technical solution to another Ø Evaluate what we asked for, not what we want to see or expect to see? Ø Guarantee every write up by every evaluator is sufficient for every requirement? Recommended Approach – Complete Evaluation of One Proposal Prior to Moving to Next One Be America’s Best…War-Winning Capabilities on Time, on Cost 29

Initial Evaluation Activities (cont. ) Team Leads: What is your preference for … Ø Collaboration among your evaluators? ü Do you want them to discuss any issues as they review? OR ü Do you want to have them wait until they have completed reviewing each offeror’s proposal? OR ü Do you want to have them collaborate at subfactor level? OR ü Do you want to wait until all offeror’s have been evaluated? OR ü Different approach? There is no one right answer! Be America’s Best…War-Winning Capabilities on Time, on Cost 30

Initial Evaluation Activities (cont. ) Team evaluators: must ensure your evaluation… Ø Does not expand scope of published criteria or change requirement Ø Considers only information contained in proposal (except PP) Ø Evaluates proposals solely against Evaluation Factor/Subfactors established in RFP (Sections L & M) and not against one another Ø Thoroughly explains in plain language, why/how proposal does or does not meet applicable requirement Ø Is consistent across all proposals Ø Is so completely documented it will withstand a protest on its own Be America’s Best…War-Winning Capabilities on Time, on Cost 31

‘Quick Look’ Status Update After receipt of proposals, SSEB Chair may update SSA with a Quick Look status brief/report Ø Provides SSA with an initial summary of proposals ü Name(s) of offerors ü Compliance of proposals with RFP ü Proper page count ü Identification of any exceptions taken to RFP requirements ü Any obvious showstoppers Recommended Practice – May be done in conjunction with SSA Training Be America’s Best…War-Winning Capabilities on Time, on Cost 32

Initial Evaluation Activities Technical Evaluations How-To Briefing for Source Selection Evaluators Be America’s Best…War-Winning Capabilities on Time, on Cost 33

Technical Evaluations Types of information you expect to receive from offerors … <Trainer to insert Section L information for Technical Factor(s)> Be America’s Best…War-Winning Capabilities on Time, on Cost 34

Technical Evaluations (cont. ) The criteria by which you will evaluate the offerors’ proposal … <Trainer to insert Section M information for Technical Factor(s) without rating information> Be America’s Best…War-Winning Capabilities on Time, on Cost 35

Technical Evaluations (cont. ) All strength(s) must be clearly documented! Ø Documentation should describe magnitude of benefit(s) to Government Ø Do. D Source Selection Procedures guide defines a strength as: “ An aspect of an offeror’s proposal that has merit or exceeds specified performance or capability requirements in a way that is advantageous to the Government during contract performance. ” Ø When a strength is identified, assess whether offeror’s proposed approach would likely cause an associated weakness (risk) which may impact schedule, cost, or performance Be America’s Best…War-Winning Capabilities on Time, on Cost 36

Technical Evaluations (cont. ) All weakness(es) must be documented Ø A Weakness is defined as a flaw in the proposal that increases the risk of unsuccessful contract performance Any/all deficiencies must be documented Ø A Deficiency is defined as a material failure of a proposal to meet a Government requirement or a combination of significant weaknesses in a proposal that increases the risk of unsuccessful contract performance to an unacceptable level Be America’s Best…War-Winning Capabilities on Time, on Cost 37

Technical Evaluations (cont. ) Technical Team Leads, must … Ø Have an approach to consolidate reviews and analysis of evaluators Ø Have a plan to reach consensus with all evaluators on each subfactor ü Do you have plan to document consensus results including disposition of any disagreement by individual evaluators? ü Consolidate reviews and analysis of evaluators? ü Have a plan to elevate disagreements when consensus is not reached Ø Review evaluator’s analysis for completeness and clarity Ø Interface with Past Performance & Cost/Price Teams Be America’s Best…War-Winning Capabilities on Time, on Cost 38

Technical Evaluations (cont. ) Team Leads: What is your process for assigning ratings? Ø Will ratings be assigned by: ü Individual evaluators? ü Team leads? ü SSEB Chair? Recommended Approach – Individual evaluators and team leads “recommend” ratings and SSEB Chair “assign” ratings Be America’s Best…War-Winning Capabilities on Time, on Cost 39

Initial Evaluation Activities Past Performance GAO case on Past Performance “Unknown” rating: Be America’s Best…War-Winning Capabilities on Time, on Cost 40

Past Performance <Trainer insert Acquisition Section L information for Past Performance Factor> What types of information do you expect from Offerors? Be America’s Best…War-Winning Capabilities on Time, on Cost 41

Past Performance (cont. ) <Trainer should insert acquisition’s Section M and discuss Past Performance evaluation factor without rating information Any questions on Past Performance criteria you are using? Be America’s Best…War-Winning Capabilities on Time, on Cost 42

Past Performance Versus Responsibility Determination: PCO Past Performance Evaluation: PPET • Does offeror have capability to perform? ü Financial capacity ü Requisite facilities, etc. ü Track record of integrity and business ethics ü Appropriate systems in place (accounting, property, quality, etc. ) • Pre-award surveys useful in providing detailed analysis • Results in PCO determination of responsibility • Identifies degree of government confidence for each competing offeror • Will offeror do the work successfully? ü Based upon its track record of recent, relevant performance • Results in Acceptable/ Unacceptable Rating or Performance Confidence Assessment Rating Be America’s Best…War-Winning Capabilities on Time, on Cost 43

Past Performance Rating Process Obtain Past Performance Information on Each Offeror and Determine Recency Assess Relevancy for Each Recent Contract Review Offeror Relevancy and Performance Ratings Assign Offeror’s PAST PERFORMANCE FACTOR Acceptable/Unacceptable or Confidence Rating Be America’s Best…War-Winning Capabilities on Time, on Cost 44

Past Performance (cont. ) Past Performance evaluators must… Ø Understand various approaches to obtain Past Performance Information (PPI) Ø Evaluate Recency of PPI ü Include explanation why each identified PPI complied with or failed to meet criterion ü Identify those not further considered Ø Evaluate Relevancy of PPI ü Use solicitation’s relevancy definitions to determine relevance for each factor or subfactor, as appropriate ü Include complete explanation why factor or subfactor was determined within selected relevancy definition ü When determined “not relevant” indicate instance not further considered Be America’s Best…War-Winning Capabilities on Time, on Cost 45

Past Performance (cont. ) Past Performance evaluators must… Ø Assess quality of information received from various sources, i. e. references/questionnaires, CPARS/FAPIIS, etc. Ø Ensure records clearly explain and support assigned rating Ø Does your documentation identify why team concurs or differs with reported performance from past performance report (CPARs, questionnaire responses, etc. )? ü While information from past performance report often forms basis of assigned performance rating, they do not have to be the same Note: Discussion and research may result in a different opinion of contract performance. When performance assessment differs from past performance report, record must document rationale for difference. Be America’s Best…War-Winning Capabilities on Time, on Cost 46

Past Performance (cont. ) Good Practices Ø Develop clear records that explain why summary Technical subfactor and Cost/Price factor (if assigned) and overall Past Performance acceptable/unacceptable rating or confidence assessments were assigned Ø Identify why team assigned various acceptable/unacceptable or confidence ratings Ø Establish rating using an integrated analysis of positive and negative performance indicators identified for Technical factor or alternatively for each subfactor and Cost/Price factor Ø Develop comprehensive observations and conclusions that support overall Past Performance factor of acceptable/unacceptable or confidence assessment rating Be America’s Best…War-Winning Capabilities on Time, on Cost 47

Initial Evaluation Activities Cost/Price Cost and Price Evaluation: Be America’s Best…War-Winning Capabilities on Time, on Cost 48

Cost/Price <Trainer insert Section L & M information for Cost/Price> What types of information do you expect from Offerors and how you plan to evaluate? Be America’s Best…War-Winning Capabilities on Time, on Cost 49

Cost/Price (cont. ) Team Lead Ø Consider topics addressed in technical evaluation charts to decide which apply to Cost/Price evaluation team Ø Consider proposal requirements and evaluation criteria outlined in Sections L & M to decide what steps must be taken to complete evaluation ü Must evaluate as outlined in Solicitation Ø Coordinate with evaluators in other areas to ensure consistency between proposed costs & other parts of proposal Be America’s Best…War-Winning Capabilities on Time, on Cost 50

Cost/Price (cont. ) Team Lead (cont. ) Ø Decide how to communicate/coordinate with other evaluation teams to gain inputs on areas that impact Cost/Price evaluation ü If using cost realism, need input from technical team to evaluate whether offeror understands requirements, and has proposed adequate material, labor, etc. Ø Coordinate documentation requirements with PCO to ensure analysis and results are properly documented GAO & Co. FC Decision Summaries – Price Evaluation (addresses Price/Cost Realism, Reasonableness) Be America’s Best…War-Winning Capabilities on Time, on Cost 51

Role of Cost/Price Team Role of Cost Team (Cost reimbursable type contracts) Ø Evaluate realism and reasonableness and develop Most Probable Cost (MPC) for each offeror Ø Assess financial executability (budget) Ø Provide cost inputs for Proposal Analysis Report, and Source Selection Decision Document Ø Document MPC for successful offeror (baseline for post award Program Office Estimate (POE) Role of Price team (Fixed Price contracts) Ø Conduct Price Analysis -- normally established by Adequate Price Competition (APC) ~ FAR preferred technique Ø Determine if prices are balanced Ø Assess financial executability (budget) Ø Provide pricing inputs for appropriate documents Be America’s Best…War-Winning Capabilities on Time, on Cost 52

Cost Evaluation Purpose – Evaluate for realism, reasonableness Ø Realism: Offeror’s proposed costs are ü Realistic for work to be performed; ü Reflective of a clear understanding of requirements; and ü Consistent with unique methods of performance and materials described in offeror’s proposal Ø Reasonableness: Price must represent to gov’t what a prudent person would pay when consideration is given to market prices Product – Most Probable Cost (MPC) for each offeror Ø SSA uses MPC in source selection decision process Ø MPC may differ from proposed cost and should reflect gov’t best estimate of contract cost Be America’s Best…War-Winning Capabilities on Time, on Cost 53

Info That May Be Used to Support Cost Evaluation Audit Reports Technical Evaluations Forward Pricing Rate Recommendations/Agreements (FPRR/A) Recent Program History Wage Determinations Cost Models Published cost/price indices Program Office Estimate Cross checks/Analogous efforts Market Information DCAA’s Supplemental Services Be America’s Best…War-Winning Capabilities on Time, on Cost 54

Cost Evaluation – Steps For Each Proposal Coordinate with Technical Evaluators Ø Understand requirements, make sure everything is proposed Ø Cross reference cost proposal against technical proposal ü Ø Is the offeror pricing what is technically proposed? Have tech evaluators conduct tech evaluations such as Do proposed hours support content and schedule? ü Are types/quantities of material, facilities appropriate? ü Materials and Subcontract Costs Ø Emphasis on high dollar items Ø Understand basis of estimate Ø How firm are proposed quotes/prices? ü Rough Order of Magnitude (ROM) versus purchase order? Were subcontracts awarded competitively? Ø What type of adjustments were taken by prime? Ø Be America’s Best…War-Winning Capabilities on Time, on Cost 55

Cost Evaluation – Rules of Thumb Always evaluate in accordance with Section M Do not use one offeror’s data to adjust another offeror’s proposal Use ENs as tool to understand discrepancies or obtain clarifications Remember you are evaluating a proposal; take into account offeror’s unique approach, cost accounting system, etc. Ø You are not developing an estimate independent of offeror’s proposal Make sure technical team’s input is adequately documented Do not write in offeror proposals, as offerors may request their proposals be returned after source selection Be America’s Best…War-Winning Capabilities on Time, on Cost 56

Price Analysis Concepts & Techniques Concepts Ø Adequate Price Competition (APC) is a price based on ü Two or more responsible offerors, competing independently; ü Priced offers that satisfy gov’t expressed requirement; ü Award made to offeror whose proposal represent best value; and ü No finding that price of successful offeror is unreasonable Ø Only one offer received…see DPAP memo, dated 27 Apr 2011 Techniques Ø Comparison of proposed prices received in response to solicitation ü APC normally establishes price reasonableness Ø Comparison of proposed prices to previous prices (same/similar items) Ø Comparison to published prices (e. g. catalog) Ø Comparison to Government cost estimates Ø Market Research Ø Analysis of pricing info Be America’s Best…War-Winning Capabilities on Time, on Cost 57

Items for Price Evaluation CLIN prices Ø Compare CLIN prices to value of CLIN effort (see section on Unbalanced Pricing) Rate evaluation (if applicable) – check rates against FPRA, FPRR, if available from DCMA Other Government Costs (as applicable) Ø Government Furnished Property/Equipment/Material (GFP/E/M) – quantify for each offeror Calculate imputed cost to treasury for contract financing, if applicable Conduct discussions where appropriate (e. g. unbalanced bids) Sum of CLINs, financing cost, OGC’s, etc. = Total Gov’t Cost Be America’s Best…War-Winning Capabilities on Time, on Cost 58

Proposal Evaluation Steps Coordinate with technical team Ø Understand requirements, make sure everything is bid Ø Is price proposal consistent with proposed technical solution? Ø Have engineers conduct tech evaluations, where appropriate Compare proposed prices received Ø Do not include non-responsible offerors Ø For wide disparities in offers, consider comparability ü Similarity of products/technical approaches ü Geographical differences ü Terms and conditions Compare proposed prices to price history for same/similar items Compare proposed prices to Gov’t Estimate and if large variance exists ask Ø What is basis of estimate and what assumptions were made? Be America’s Best…War-Winning Capabilities on Time, on Cost 59

Unbalanced Pricing Exists when price of one or more CLINs is significantly overstated (despite acceptable total evaluated price) Ø Typically occurs in conjunction with under pricing of one or more line items Ø May increase performance risk Ø Does not automatically result in rejection of offers How do you know it is present? Ø Comparison of each offeror CLIN prices – are there significant differences? Ø Comparison of each offeror CLIN prices to Gov’t estimated CLIN prices – are there significant differences? Mathematically unbalanced – offer is based on nominal prices for some work and enhanced prices for other work Give offeror opportunity to address your concerns Be America’s Best…War-Winning Capabilities on Time, on Cost 60

Initial Evaluation Activities Contracting Be America’s Best…War-Winning Capabilities on Time, on Cost 61

Contracting Team must… Ø Determine if proposal conforms to all Terms & Conditions and identify any exception(s) taken Ø Ensure all required certifications Ø Verify that proposal meets all technical requirements and identify any exception(s) taken Ø Capture any offeror assumptions Ø Identify discrepancies in proposal Ø Determine how to capture proposal strengths in contract Ø Prepare official contracting evaluation record Be America’s Best…War-Winning Capabilities on Time, on Cost 62

Initial Evaluation Activities Evaluation Notices Be America’s Best…War-Winning Capabilities on Time, on Cost 63

Evaluation Notice (EN) ENs are tool to conduct exchanges with offerors for purposes of Clarifications, Communications, or Discussions (see next chart) Exchanges with offerors should: Ø Identify type of exchange Ø Clearly classify ENs that result from deficiencies Ø Be specific to individual concerns ü Writing single EN for multiple topics not recommended ü Crossing subfactors not appropriate ü Easier to discuss and disposition “Item Specific” issues Ø Be prepared by SSEB for review by PCO and Legal Counsel Be America’s Best…War-Winning Capabilities on Time, on Cost 64

Types of Exchanges Clarifications* As needed with Award Without Discussion (AWOD) Communications* To determine competitive range Shall Discussions/Negotiations Conducted with offerors in the Competitive Range Adverse Past Performance Goal is to obtain best value Information (PPI) is Discuss deficiencies, Adverse past determining factor in exclusion weaknesses, and other performance, not aspects of proposal to May previously addressed enhance award Inclusion or exclusion in Adverse past performance Past Performance Competitive Range is information to which offeror Information relevance uncertain has not had opportunity to respond Enhance Government Minor or clerical errors understanding of proposal Discuss efforts above mandatory minimums Facilitate Government * Neither Clarifications evaluation process nor Communications Conducted either orally or in writing or both allow for a Proposal Address ambiguities, Revision perceived weaknesses, Areas may also include issues of compliance with omissions, mistakes, etc. RFP other than evaluation factors GAO Cases on Clarification, Communication & Discussion: Be America’s Best…War-Winning Capabilities on Time, on Cost 65

EN Guidelines Write an EN for each weakness and deficiency at same time you complete Evaluation Worksheet Ø Use EN writing formula (on next chart) Address only one issue per EN Don’t ask for more than originally asked for or need – use proposal instructions as your guide Be clear, complete, and direct when you write your EN Ø Part of process is for offerors to improve their proposals Ø Do not ask offerors to be mind readers EZ Source EN Template Be America’s Best…War-Winning Capabilities on Time, on Cost 66

EN Writing Formula – Key to a Good EN Write to the offeror … use words like “Your proposal states … “ Ø In 1 st paragraph, start with a very succinct description of the issue, e. g. “Your proposal does not provide sufficient information IAW the proposal instructions. ” OR “Your approach to software development appears risky. ” ü Be sure to reference and quote applicable parts of offeror’s proposal and applicable parts of RFP, e. g. Section L Proposal Instructions and/or SOW/Spec paragraph numbers Ø In 2 nd paragraph, further elaborate on the issue if needed, clearly explaining issue that is associated with applicable parts of proposal referenced (e. g. missing information, need for further explanation, confirmation of your understanding of proposal, need for clarification) Ø In 3 rd paragraph, politely request the information you want (Please provide the information, Please explain … clarify …) Be America’s Best…War-Winning Capabilities on Time, on Cost 67

<Trainer to insert the latest template that is located in the AFFARS Library under 5315, 3> Be America’s Best…War-Winning Capabilities on Time, on Cost 68

EN Development Hints Refer back to a requirement; reference requirements as specifically as possible Do not ask leading questions Do not provide solutions or allude to another offeror’s solution “Never” ask why an offeror did not do something as part of their approach unless they did not address a specific requirement Be consistent from one offeror to the next; use same words for same issues Remember non-technical personnel must be able to understand Ø Do not use jargon or acronyms unless defined or generally used Be America’s Best…War-Winning Capabilities on Time, on Cost 69

ENs and Proposal Updates Request offerors to provide Proposal Change Pages along with their EN responses, for anything that will be incorporated into the contract EN responses can also substitute for actual Proposal Change Pages when information affected will not specifically be part of contract Offerors continually update proposals; evaluators always have access to up-to-date proposals Ø Minimizes efforts for both offerors and Government when addressing Final Proposal Revisions Ø Ensures no surprises after discussions close Be America’s Best…War-Winning Capabilities on Time, on Cost 70

Initial Evaluation Activities Evaluation Notice (EN) <Trainer to provide samples of various ENs that shows the differences between a Clarifications EN, Communications EN or Discussions EN> When reviewing ENs ask: Is it well written? If not, how would you improve? Be America’s Best…War-Winning Capabilities on Time, on Cost 71

Be America’s Best…War-Winning Capabilities on Time, on Cost

Award Without Discussions (AWOD) AWOD occur only in “limited” circumstances for solicitations at > $100 M Ø Solicitation must state that Government intends to evaluate proposals and make AWOD Ø Award made to proposal(s) that represent best value after evaluation IAW factors and subfactors in solicitation Ø If Government later determines it is necessary to conduct discussions, rationale shall be documented in contract file Ø Clarification Exchanges are permitted when conducting AWOD Must obtain contract clearance and SSA approval SSAC chair (SSEB chair if no SSAC) will provide a recommended source to SSA Ø Provide minority opinion, if applicable Be America’s Best…War-Winning Capabilities on Time, on Cost 73

Establish Competitive Range is comprised of all most highly rated proposals Ø May recommend elimination of one or more offerors Ø May be reduced for purposes of efficiency, if so stated in RFP Competitive range must be initially established prior to discussions Ø Subsequent to initial competitive range and conduct of exchanges, if an offeror’s proposal should no longer be included, proposal shall be eliminated from consideration for award When establishing competitive range, each proposal must be rated against all evaluation criteria SSEB prepares Competitive Range briefing if required by SSA Ø Present initial evaluation results for all factors/subfactors Ø Brief all strengths, deficiencies and weaknesses Ø May include interim ratings for release during discussions Be America’s Best…War-Winning Capabilities on Time, on Cost 74

Competitive Range Elimination Notification Promptly issued written notice (initial or subsequent) to all unsuccessful offerors Ø Notice must state basis of determination Ø Inform that a proposal revision will not be considered Ø Notify offeror(s) that they may request a debriefing Offeror may request delay of pre-award debriefing until after award Ø If delayed, include all information normally provided in a postaward debriefing Be America’s Best…War-Winning Capabilities on Time, on Cost 75

Be America’s Best…War-Winning Capabilities on Time, on Cost

Pre-FPR Evaluation Activities – Conduct Discussions Objective – Maximize gov’t ability to obtain best value Release interim ratings (may be accomplished at initiation and conclusion of discussions) Ø PCO shall provide offerors all of their own initial ratings and results of initial pricing analysis Ø May use actual rating charts briefed to SSA Release ENs Ø May hold separate face-to-face meeting with each offeror to ensure ENs are understood before offeror responds Ø Receive and evaluate response to ENs from offeror(s) Ø Follow up may be written ENs or face-to-face discussions Be America’s Best…War-Winning Capabilities on Time, on Cost 77

Conduct Discussions (cont. ) Release ENs (cont. ) Ø Resolve issues Ø Evaluate all additional responses received Ø EN responses to weaknesses may result in deficiencies Ø Must be identified to offeror when discovered Ø Revise all evaluations, model contract and update records Ø Prepare pre-FPR evaluation results Ø Conduct MIRT and/or Peer Review, if required Ø Obtain legal review, FPR contract clearance, and SSA approval Ø Release any revised interim ratings ü Updated ratings may be disclosed to offerors prior to request for FPR ü Issued to all offerors in Competitive Range • May release draft to ensure understanding of all issues Be America’s Best…War-Winning Capabilities on Time, on Cost 78

Conduct Discussions (cont. ) May include bargaining (persuasion, alteration of assumptions and positions, give-and-take) that applies to: Ø Price, Ø Schedule, Ø Technical requirements, and Ø Type of contract, or other contract terms Evaluation credit offered in solicitation for technical solutions exceeding mandatory minimums - PCO may Ø Negotiate for increased performance (strength) Ø Suggest removing strengths to make proposals more competitive Ø Consider and discuss if offeror’s strength causes associated weakness (risk) Be America’s Best…War-Winning Capabilities on Time, on Cost 79

Conduct Discussions (cont. ) Government personnel shall not do the following: Ø Favor one offeror over another Ø Reveal an offeror’s technical solution or intellectual property Ø Reveal an offeror’s price Ø May inform offeror that price is too high or too low, or what is reasonable Ø Identify names of individuals providing past performance information Ø Knowingly furnish source selection information Be America’s Best…War-Winning Capabilities on Time, on Cost 80

Concluding Discussions Once offerors have had a reasonable opportunity to address the ENs and SSEB has a clear understanding of what each offeror is proposing, Ø SSEB provides a Pre-FPR brief to the SSAC or SSA on results of discussions Ø PCO notifies offerors that discussions are closed and issues a request for Final Proposal Revisions (FPRs) Be sure you have a clear understanding of proposals and require no additional information Ø Once PCO closes discussions and asks for a FPR you generally cannot contact offerors until award decision Be America’s Best…War-Winning Capabilities on Time, on Cost 81

Request for Final Proposal Revision (FPR) Issued at conclusion of discussions Ø Gives offeror opportunity to submit a FPR Establishes common cut-off date Proposal revisions form baseline for Final Evaluation Ø Technical Baseline Ø Terms and Conditions Ø Business Arrangement All Changes Should Be Traceable To Original Proposal Be America’s Best…War-Winning Capabilities on Time, on Cost 82

Be America’s Best…War-Winning Capabilities on Time, on Cost

Final Evaluation Activities Proposal Evaluation SSEB evaluates Final Proposal Revision Ø Reassesses ratings including Strengths, Deficiencies, Weaknesses relative to any proposal revisions Ø Presents results of evaluation to SSAC, when used, and SSA Documents final Proposal Analysis Report (PAR) SSAC conducts and documents Comparative Analysis Report with decision recommendation Ø If SSAC is not used, SSEB will perform if requested by SSA SSEB begins preparing final Decision Briefing PCO obtains legal review and contract clearance from Clearance Approval Authority Official Record: PAR & Comparative Analysis Report Be America’s Best…War-Winning Capabilities on Time, on Cost 84

Be America’s Best…War-Winning Capabilities on Time, on Cost

Considerations for SSAC’s Comparative Analysis Report (CAR) with Source Selection Recommendation = main product of SSAC performs comparative analysis and do not base recommendation entirely on ratings -- they are only indicators Ø When comparing proposals carefully consider ü Factor and Sub-factor Rating/Risk ü Most important, narratives associated with strengths, weaknesses and deficiencies from the SSEB PAR and Final Evaluation Briefing ü Significance of each strength and weakness (not number) ü Order of importance of Evaluation Factors and Sub-factors ü Emphasis areas, thresholds and objectives ü Merit of paying a premium for perceived benefits Ø Tradeoffs need not be quantified (FAR 15. 308) Be America’s Best…War-Winning Capabilities on Time, on Cost 86

Comparing Benefits of Proposals Discussion at SSAC meeting should help make the CAR sound as specific as necessary Need to be sure assessments are consistent with evaluation factors and requirements Discussions should address why one proposal is valued over others Integrity - Service - Excellence 87

Decision Briefing Must include sufficient detail narrative descriptions of each offeror’s strengths, deficiencies, weaknesses, and PP Ø Ø Ø Relevant charts from Competitive Range Briefing Final Ratings for Technical/Risk, if applicable Past Performance Rating, if applicable Cost/Price Resolution of unresolved disagreements (if any) between evaluators to ensure SSA is in the know on what they were & how they were addressed before making SS decision Ø Contracting considerations and any exception to T&C’s Ø A source selection recommendation Ø Summary Chart Part 5315 of the AFFARS Library provides a tailorable template for a Final Decision Briefing to SSA Be America’s Best…War-Winning Capabilities on Time, on Cost 88

Select Source Based on a comparative assessment of proposals against all source selection criteria in solicitation Decision shall represent SSA’s independent judgment Document source selection decision in Source Selection Decision Document (SSDD) Ø Include rationale for any business judgments and tradeoffs made or relied on by SSA Ø Include benefits associated with additional costs Ø Fully releasable to GAO and others – proprietary information redacted if not authorized Official Record: SSDD Be America’s Best…War-Winning Capabilities on Time, on Cost 89

Be America’s Best…War-Winning Capabilities on Time, on Cost

Award & Debriefings Activities Synopsis of Contract Award Announcement Ø Prepare 1279 Report (>$6. 5 M) Notification to Unsuccessful Offeror(s) Pre-award Notice of Exclusion from Competitive Range Ø Pre-award Notice for Small Business Programs Ø Post-Award Notice Ø Ø Upon notice of award, unsuccessful offerors have 3 calendar days to request a debriefing and 10 calendar days to file a protest (or 5 days after debriefing) Award to Successful Offeror Ø Successful offeror may also request a debriefing Be America’s Best…War-Winning Capabilities on Time, on Cost 91

The Debriefing Government must conduct if: Ø Offeror makes written request Ø Request is received within 3 calendar days after offeror received notice of exclusion from competitive range, or notice of contract award Government may consider untimely requests Debriefing should be conducted within 5 days or as mutually agreed Debriefing Offerors Official Record: Debriefing Documents Be America’s Best…War-Winning Capabilities on Time, on Cost 92

Purpose of Debriefing Explain rationale for Government’s decision Ø Elimination from competitive range Ø Offeror was successful or unsuccessful Instill confidence that offeror was treated fairly Assure offeror that proposals were evaluated IAW RFP, laws, and regulations Identify proposal short-comings so offeror can better prepare in future Ø Both content and administrative issues Reduce misunderstandings and protests Provide opportunity for feedback Possibly minimize protests Be America’s Best…War-Winning Capabilities on Time, on Cost 93

Participation in Debriefings PCO chairs debriefings supported by SSEB and appropriate Evaluation Team members Thorough preparation by all participants is essential to success Ø Review proposal, Evaluation Worksheet(s) ENs, EN responses, and any proposal changes ü Review SSAC Briefing slides you may need to address Ø Debriefing material should be outlined, coordinated by JAG, and strictly followed during debriefing ü Be clear on who is going to say what when: PCO, SSEB Chair, Team Leader, you Ø Follow direction of PCO, SSEB Chair, Team Leader Face-to-face debriefings are most effective, however PCO may choose other methods, e. g. , via phone or videoconference Be America’s Best…War-Winning Capabilities on Time, on Cost 94

What to Debrief Introduction Purpose of debriefing Ground rules and agenda Source Selection Process Evaluation Factors/Subfactors Evaluation Results for Offeror’s proposal Rationale for Eliminating Offeror (pre-award) Reasonable responses to relevant questions Avoid allowing additional questions after debriefing Do Not discuss proprietary information of competing offerors Official Record: Debriefing Documents Be America’s Best…War-Winning Capabilities on Time, on Cost 95

Records Retention Source Selection Documentation Guidance GAO Protests: How to Properly Draft CO’s Statement of Facts Be America’s Best…War-Winning Capabilities on Time, on Cost 96

SS Records & Retention Ensure team has written operating procedures, i. e. ROEs regarding protection, management and disposition of Source Selection records Follow Do. D and SAF/AQC guidance on records retention Use electronic tool to extent possible and maintain in format that facilitates discovery (PDF) E-mails and working papers constitute SS documents Do not destroy working papers until all relevant information is captured in official record Consult legal counsel prior to deletion/destruction of any SS documents Be America’s Best…War-Winning Capabilities on Time, on Cost 11/1/2020 97

Ethics And Procurement Integrity Use this time to have legal conduct briefing and to get Source Selection team to sign Non-Disclosure Agreement or Conflict of Interests, etc. Be America’s Best…War-Winning Capabilities on Time, on Cost 98

What is Source Selection Information (SSI)? SSI means ten types of information (FAR 2. 101) 1. Bid prices submitted by bidders 2. Costs or prices submitted by offerors 3. Source selection plans 4. Technical evaluation plans 5. Technical evaluations of proposals 6. Cost or price evaluations of proposals 7. Competitive range determinations 8. Rankings of bids, proposals, or competitors 9. Reports and evaluations of SSAC 10. Other information, if PCO determines that disclosure would jeopardize integrity of procurement, and is marked “Source Selection Sensitive – See FAR 2. 101 & 3. 104” Be America’s Best…War-Winning Capabilities on Time, on Cost 99

REFERENCES Do. D Source Selection Procedures Ø Effective 1 Jul 2011 and mandatory for all competitive acquisitions utilizing FAR Part 15 procedures Source Selection Trainer’s Reference/Resource Material Ø Assists trainers in presenting SS Phase I & II training modules Types of Contracts - FAR Part 16. 103 & 104, DFARS 216. 1 Contracting by Negotiation - FAR Part 15, DFARS 215 Acquisition of Commercial Items - FAR Part 12. 303 Describing Agency Needs (Rqmts) - FAR Part 11. 101, DFARS 211. 1 Market Research - FAR Part 10, DFARS 210 Acquisition Planning - FAR Part 7. 1, DFARSS 207. 1 Conflicts of Interest - FAR Part 3. 104, DFARSS 203. 104 Be America’s Best…War-Winning Capabilities on Time, on Cost 100

Summary – You Can Have A Major Impact Direct impact on selecting Best Value Ø Warfighters live with the result Bolster our position in event of protest High stakes “game” Ø Severe consequences if we don’t do it right Not luck but Ø Good disciplined process throughout Ø Hard work Ø Expertise, dedication and concentration Be America’s Best…War-Winning Capabilities on Time, on Cost 101

Summary of Changes Phase II Source Selection Execution Be America’s Best…War-Winning Capabilities on Time, on Cost 102

Linked Charts Be America’s Best…War-Winning Capabilities on Time, on Cost 103

Proposal Analysis Report (PAR) Documents results of Source Selection Evaluation Board’s (SSEB) evaluation Signed by SSEB Chair and Contracting Officer Includes: Assessment of Cost/Price Past Performance (unless waived) Technical If SSAC is NOT used, SSA may request SSEB to provide Source Selection Recommendation (including any minority opinion, if applicable) Guidance contained in MP 5315. 3 Official Record: PAR Be America’s Best…War-Winning Capabilities on Time, on Cost 104

Comparative Assessment Report Access to Comparative Assessment Report and Recommendation Template can be found on AF Contracting Knowledge under AFFARS Library, Part 5315. 3 - Community Advice. Be America’s Best…War-Winning Capabilities on Time, on Cost 105

106

Source Selection Decision Document (SSDD) SSDD… Shall reflect SSA’s independent, integrated, comparative assessment Shall include SSA’s rationale for any business judgments-tradeoffs Included in the official Source Selection file Fully releasable to the GAO and others Proprietary information redacted if not authorized Must consistently track to Requirements, Evaluation Factors, Decision Briefing and PAR Compares Proposals by Factors/Subfactors SSDD should limit source selection sensitive information to only information pertinent to the decision. Be America’s Best…War-Winning Capabilities on Time, on Cost 111