BEST PRACTICES KARI SANTORO Branch Chief Assisted Acquisition

BEST PRACTICES KARI SANTORO • • Branch Chief – Assisted Acquisition Services South Team GSA

U. S. General Services Administration Best Practices: Evaluating Offers From the Technical Side presented by Kari Santoro

Background R 03 Streamlining Processes & Techniques since FY 2007 Purpose: Receive higher quality contractor proposals Increase ability to award on initial submissions Avoid use of most FAR part 15 processes on Task Orders Save time Reduce unnecessary work • Reduce chances of protest 2

Simplify RFQs/RFPs • Prepare clear, concise instructions • Select evaluation criteria based on what’s most important • Simplify documentation: – – – Use page limits Include notes to promote understanding Prevent contractor’s use of caveats and assumptions Provide templates (e. g. , staffing matrix, past performance form) Use “Key Personnel Qualification Matrices” in lieu of resumes or overly prescriptive personnel qualifications • Use pre-quote/pre-proposal conference to set ground rules & ensure understanding • Use “cut-off” dates for questions • Post answers to questions promptly to keep timeline on track • Easier & more logical process flow • Ensures contractors better understand requirements • Allows the Government to clarify instead of entering into discussions • Enables Government to award on initials without a prolonged negotiation process • Prevents unnecessary work • Cuts the chances of a protest and helps you to better defend against one if you do get one 2

Selection of Technical Factors • Look at the RFP & PWS holistically. • Avoids unnecessarily complicated evaluation criteria • Use as few technical factors as possible as a general rule • Reduces the time it will take to evaluate • - Select criteria that is Distinguishing & Meaningful • Condenses the amount of information contractors are required to furnish allowing more timely responses - Consider requirements and risks in developing relevant evaluation criteria • Allows the evaluation to concentrate on the most significant elements No need to “re-qualify” companies that already have proven capabilities in a given technical discipline. - GWAC & Schedule holders have a record of successful performance. - Rather than focusing on evaluating already proven expertise, select factors that are tailored to discriminate which company is best to satisfy requirements. • Yields more meaningful information on which to base a best value decision 2

Sample - Selection of Technical Factors 2

The Offerors proposal shall be evaluated to assess its clear understanding of the PWS requirements, demonstrated technical and management experience and the soundness of its approach to: a. Use Agile principles for project management and software development, and its approach to perform, manage, define, quantitatively manage, and optimize software development (as defined in CMMI model), and to ensure software development processes to be used on the Task Order provide an integrated approach across the enterprise to improve quality and performance, while reducing redundancy, complexity, and cost. b. Design, develop, upgrade, migrate/port, test, debug, continually improve, optimize, integrate, and operate/sustain high quality sites, software, communities, applications, systems/subsystems with and without backend workflows for program and process management, managed role-based access for editing, content contributions and maintenance, proactive compliance with appropriate laws, policies, and regulations as identified in the PWS; and promotion of improved content discovery, device agnostic mobile usage, user engagement and enhancement of the overall user experience for Share. Point sites. c. Use of current and emerging techniques, technologies, tools, processes, and to deliver custom-built solutions. Working with various levels of internal and external customer teams to define requirements, innovatively develop affordable and easily sustainable high quality IT solutions. Discuss processes to ensure stakeholder customer satisfaction, manage expectations, control requirements creep, monitor schedules during development and maintenance, measure performance and continually improve IT solutions. Use of prototypes to evaluate untried, high-risk, innovative improvements. Capability to gain approval for use of cloud technologies. TECHNICAL APPROACH The Offeror shall present a technical approach that: a. Describes how your software development processes provide an integrated approach across the enterprise to improve performance while reducing redundancy, complexity, and cost. b. Describes how you will design, integrate, and sustain Share. Point sites which provide improved content and enhances the overall user experience. c. Describes how you will create an innovative knowledge management repository that integrates discovery, delivery, and management of learning assets to produce a valuable usercentric experience. d. Describes your approach for applying thought leadership and collaborating with internal and external stakeholders to: (1) identify and define IT requirements. (2) develop a plan of action and schedule. (3) manage stakeholder expectations. 2

Demonstrated Management and Staffing Experience, Clear Understanding of Requirement, and Soundness of Approach The Offerors proposal shall be evaluated to assess its clear understanding of the PWS requirements, demonstrated management, organizing, and staffing experience and soundness of approach, as proposed in its template, qualifications in the resume submissions for Key Personnel with signed letters of intent, draft Program Management Plan (PWS 6. 1), to include addendums for Quality Control, Risk Management (PWS 5. 1), and Transition Plan (PWS 5. 2) a. Manage, organize and staff its team, to include subcontractors, and to transition and accomplish the broad scope of requirements in the PWS; clearly assign and delineate the roles and responsibilities and lines of authority. Balance corporate requirements with productive hours on the Task order. Assessment of organizational conflicts of interest (OCIs) and plan to eliminate or mitigate OCIs. Implement its company’s policies for normal work week, overtime and comp-time, uncompensated overtime, timecards, approvals for use of overtime or comp-time, telework and leave. b. Plan, budget, schedule, and deliver projects and work tasks, to include TDL work effort (PWS 6. 3), allocation of resources and assignment of tasks to ensure staff are qualified, experienced, and matched to assigned tasks to be performed, and to promote staffing optimization and quality performance. c. Select, hire, on-board, retain, replace and motivate qualified and experienced key personnel and staff, and to maintain technical currency and professional development. Ability to attract, quickly hire/acquire, on-board, and retain qualified personnel, consultants, or subcontractors as surge during high demand or short duration periods, such as for the base period of the contract; or for specialty skills, such for innovation and prototyping activities. d. Management approach to provide quality products, services and deliverables, and self-inspecting the quality, timeliness, responsiveness, customer satisfaction, and performance requirements to meet, track, and report on the quality performance standards in Attachment C, Quality Assurance Surveillance Plan. Ability to plan, direct, control, manage, and monitor personnel, subcontractors, and tasks across the entire contractual effort to promote and measure innovation and continual improvement; reduce and manage risk; ensure quality customer service; monitor and control quality, schedules, and costs; identify problems and implement corrective solutions/conflict resolutions; promote collaboration and communication; team with Government and other support contractors to achieve contract objectives; ensure its team understands and complies with the contract. STAFFING FACTOR The Offeror shall provide the following: a. Staffing Matrix: Submit your proposed staffing on the template furnished with the RFP. b. Staffing Narrative: Provide rationale for your proposed labor mix in a written narrative that: 1) Explains how your team will be organized. 2) Describes the alignment of the roles and responsibilities among proposed positions and how you will leverage your staff. 3) Describes how your team will recruit and retain personnel with the specialized skills needed. c. Key Personnel: Explain the rationale behind your selection of the key personnel proposed for the positions listed below. Describe each individual’s relevant qualifications and experience in leading staff in an IT environment of similar complexity to the requirements in this PWS: 1) Program Manager 2) Lead Applications Developer/Architect 3) Site Administrator 2

Prepare for the Technical Evaluation • Use a streamlined technical evaluation should position Govt. to award on initials. However: v In solicitation, reserve Govt’s right to establish a competitive range and enter into discussions v A submission that contains material defects that increases risk to an adverse level must be corrected before award. • • At the start of the Evaluation Phase: – Brief evaluation team on procurement integrity responsibilities – Vet evaluators for Conflicts of Interest – Get signed Non-Disclosure Agreements (NDA) – Ideally, ensure each member of the evaluation team is available to dedicate their time to the technical evaluation during evaluation phase. Set the stage for conducting the Technical Evaluation: – Establish a Simple Rating Scheme - Serves as basis for rating each Technical Factor - Recommend using an adjectival rating scheme, such as: v Exceeds, Meets, Does Not Meet v Excellent, Very Good, Unacceptable 2

Technical Evaluation – Establish common understanding of terms: - Strengths = Positive attributes in the offeror’s submission - Weaknesses = Shortcomings (i. e. , flaw) in the offeror’s submission - Risks = Event that may occur. Consider likelihood of occurrence & consequence or impact - Deficiency = Material failure to meet a requirement – Establish evaluation methods for differentiating among submissions. For example: - If “Exceeds” rating is assigned, list distinguishing quality or characteristic of submission that contributes to rating. State why such quality/characteristic is of value to the Government. - If “ Does Not Meet” rating is assigned, identify the “material defects” in the submission that increase the likelihood of failure. 2

Technical Evaluation Workbook • Use “Technical Evaluation Workbook” to capture evaluator comments – Includes the adjectival rating scheme and definitions – Rephrases the evaluation criteria back to the technical evaluation team in the form of an openended question – For each question, request a narrative explaining the evaluator’s affirmative or negative response • Workbook helps keep evaluators focused • Encourages a solid explanation behind each factor’s rating Consensus • Conduct a consensus discussion with the evaluation team o Agree upon the rating on each factor for each offeror o Identify the key reasons for each assigned rating 2

Close-out Evaluation Activities • Remind evaluators of duty to maintain integrity of procurement process. o Protect proprietary and acquisition sensitive information. o • Tips for Writing Consensus Report Prepare a single Technical Evaluation Consensus Report – • Collect all workbooks, proposals, and evaluation materials at the conclusion of the evaluation. Ensure evaluation is conducted i. a. w. criteria contained in RFQ/RFP Limit how much extraneous information is put into technical evaluation – Minimize “background” information – Use bulletized lists to show salient information • Address elements of the factor in same sequence as they appear in RFQ/RFP • Ensure evaluation addresses everything required in the technical evaluation criteria – Use language that matches (i. e. , “supports”) the assigned ratings – Use language that is consistent with the definition of assigned ratings • Avoid reference to price/costs • Spell out all acronyms on first use 2

Streamlining Techniques • Consider use of “Interview-style” Orals 2

Interview-style Orals • Just Like a Job Interview! o Government asks questions o Contractor answers • Contractor talks. • Government evaluators listen and write notes. • It is not a discussion. 2

Interview-style Orals – Benefits • Get Price, Staffing, and Past Performance in Written Submission • Identify which Technical Factors will use Interview-Style Orals • Government develops Core Questions for each Technical Factor • Contractors knows Technical Factors will be oral but not the questions • • o Develop clear, concise questions related to each factor o Use as few questions as possible o Generally avoid overly complex questions Two sessions: o Core Session o Clarification Session (if needed) Sessions are timed and controlled by the Contracting Officer 2

Sample Interview-style Orals Questions Technical Approach 1) Where do you see opportunities to provide innovative solutions? Your answer should include your approach to take advantage of said opportunities. 2) Under Task 1 you are to take responsibility for an existing system/application which requires enhancements and additional development. Explain your processes to take the project from acceptance to final solution. Focus the answer on the engineering processes and specific deliverables necessary to manage the solution, control change, and transition to Operations and Maintenance. 3) Within the next few years, the United States Army Europe will undergo significant transition in both mission and manning. You will be asked to keep up with emergent mission technologies and provide unique and complex solutions. Many times, these integrations have never been done before. a. How will you prepare your team with the knowledge and skills required to implement these technologies? b. How will you fill any gaps of current skills within your team? c. How does your team stay in line with the U. S. Army Technology Roadmap? 2

Sample Interview-style Orals Questions Staffing 1) Describe your approach to staffing concurrent missions in garrison and at austere locations. Task Understanding 1) How is your company prepared now (today) to successfully perform the requirements of this task on day one of operations? 2) What risks does your company foresee arising out of the performance of this task and how will you manage those risks? 3) How will your company handle shifting priorities in a timely manner during task performance, and how will it transfer knowledge to federal employees and new company employees? Quality 1) What aspects of quality control will your company implement to meet the performance metrics outlined in the PWS? 2) How will your company sustain quality control throughout the performance of this task? 2

Sample Interview-style Orals Questions Sample Scenario-based Core Question: SCENARIO: On Monday when the Service Desk is being flooded with complaints of severe lag time you also learn a key project engineer that is implementing the IPv 6 project and your Desktop Support Lead have accepted positions at another company and will be leaving in two weeks. Then, two weeks later the COR brings to your PM complaints from several tenant organizations that delays in completing desktop support trouble tickets is affecting their ability to perform their mission. Given the scenario above, describe: 1. What key issues would you focus on? 2. How would you prioritize your actions? 3. What corrective action steps would you take to address the situations? 2

GSA Contacts • For additional information, contact: • Kari Santoro, Branch Chief, Assisted Acquisition Service: o Kari. Santoro@gsa. gov o 215 -446 -5831 2

2

LEADING THE ACQUISITION WORKFORCE PAUL Mc. DERMOTT • • Former GSA Regional Commissioner Retired

CAREER MGT TIPS • Support a CUSTOMER; solve a PROBLEM; assist a CO-WORKER; develop a new IDEA; improve a PROCESS. • Be the person that rises to challenges and overcomes obstacles. • Be someone who has initiative. • Hold yourself to a higher standard than those around you. • Even if your boss is a jerk and your company is run by a bunch of idiots, it is your career. You need to ask yourself regularly what you have positively contributed to the organization mission. You never need permission to do great work. • Always spend qualitative time thinking of the unintended consequences of your • Everyone runs into roadblocks on occasion; figures don’t match, you can’t find the right data to back up your thesis, someone is unwilling to cooperate with you. That’s normal. But leadership doesn’t want you to come and tell them what problems you have. They want you to come to them with a list of proposed solutions and which solution you prefer and why and give them a road map to achieve it. • Be the type of employee that co-workers are thrilled to work with because of your values (e. g. respect, integrity, collaboration, positive attitude, etc ). You are the person your colleagues are happy to see at work each day. • Have a mentor(s)!