In Depth Panel Review Training Activity Mock Panel

  • Slides: 20
Download presentation
In Depth Panel Review Training

In Depth Panel Review Training

Activity: Mock Panel Review To evaluate the Need for Assistance, reviewers will consider the

Activity: Mock Panel Review To evaluate the Need for Assistance, reviewers will consider the extent to which the application includes: 1. A concise problem statement that identifies the current condition(s) to be addressed by the project 2. Supporting information or data detailing the scope and nature of the problem 3. Current challenges standing in the way of addressing the problem 4. A clear description of the community to be served and who the intended beneficiaries are What score, out of 10 possible points, did you give the application narrative? Why?

Panel Review Roles Review Director: oversee the entire panel review process Logistics Contractor: handles

Panel Review Roles Review Director: oversee the entire panel review process Logistics Contractor: handles payment and ARM support Program Area Manager (PAM): An ANA representative with the responsibility to approve the PSR Subject Area Manager (SAM): An ANA representative charged with assisting the panel on the process and the editing of the PSR Reviewer: A member of the panel charged with evaluating the application and documenting their evaluation by scoring and commenting on the application strengths and weaknesses Facilitator: A member of the panel charged with managing the panel discussions and writing the PSR

Panel Review Structure Facilitator 1 3 Reviewers Facilitator 2 3 Reviewers SAM PAM SAM

Panel Review Structure Facilitator 1 3 Reviewers Facilitator 2 3 Reviewers SAM PAM SAM Panel Review Director SAM PAM SAM

Approval Process of PSRs

Approval Process of PSRs

Calendar of Events Pre-Panel (April 1 st – April 27 th) Reviewers and facilitators

Calendar of Events Pre-Panel (April 1 st – April 27 th) Reviewers and facilitators are identified and trained Session 1: P&M, EMI, and ERE (April 25 th – May 9 th) Reviewers and facilitators evaluate and panel applications Session 2: SEDS and SEEDS (May 9 th – May 23 rd) Reviewers and facilitators evaluate and panel applications Post Panel Payment is sent to reviewers and facilitators

The Application Review Module (ARM) Do ü Enter the Score First ü Use the

The Application Review Module (ARM) Do ü Enter the Score First ü Use the ARM text box to write your comments ü Spellcheck ü Save every 20 -30 minutes Do Not × Use Google Chrome × Click the ARM support link on the ARM login page Email ANAReviewer@acf. hhs. gov with all ARM questions!

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by facilitator & submitted to SAM/PAM Three Principles for Reviewing Applications ü Evaluate the extent to which the application responds to the FOA Evaluation Criteria ü The FOA Evaluation Criteria is the only measure used to evaluate the application ü Do not compare applications against each other

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Narrative Descriptor Score Excellent 93 -100 Very Good 86 -92 Good 78 -85 Fair 70 -77 Needs Significant Improvement 0 -69

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM How Do You Determine the Significance of a Weakness and a Strength?

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Examples of Good Weakness Comments The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach) The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be evaluated, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected) The applicant did not describe anywhere in the application how community input was used in the development of the project, making it unclear if the project proposal is truly community-driven. (page 25, Approach)

Reviewers read application & evaluate using FOA criteria Not everyone agrees at the beginning

Reviewers read application & evaluate using FOA criteria Not everyone agrees at the beginning Hearing the other reviewers point of view made me reconsider my score Reviewers write comments & determine preliminary score Panel members discuss applications Structured and focused What is a panel discussion like? Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Initial scores vary. Sometimes by 20 points or more We spent about an hour talking about each application

Reviewers read application & evaluate using FOA criteria Panel Group Norms • Attend all

Reviewers read application & evaluate using FOA criteria Panel Group Norms • Attend all meetings • Keep Focused • Listen and show respect Share initial reviewer comments Reviewers write comments & determine preliminary score Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Panel members discuss applications The FACILITATOR sets group norms, focuses the meeting, asks probing questions, shares information, and documents the discussion Focus the meeting by creating a meeting plan • • • Probing Questions What criteria did you use? How did you determine this score? What documentation did the application provide that substantiates the approach? Document the discussion to help write the Panel Summary Report

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Requirements of a Panel Summary Report (PSR) 1. Factually Accurate 2. Adheres to the FOA Evaluation Criteria 3. Consistent 4. All comments justify the score 5. Polished and Well-Written

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Example of an Inconsistency in a PSR Strength Comment The applicant clearly showed how the current staff will manage and complete the project on time. (page 75, Objective Work Plan) Weakness Comment The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach)

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Writing a PSR that Represents the View of the Panel ü Create an initial PSR ü Take notes during the panel discussion ü Draft the final PSR ü Submit the final PSR to the PAM/SAM for comments and approval ü Share the final PSR with the reviewers

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Tips on Writing a PSR that will be Approved ü Reference Application Page Numbers ü Use the wording of the relevant FOA evaluation criteria ü Make it grammatically correct

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary

Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Example of a Good PSR Comment The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be measured, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected) FOA Text in Outcomes Expected Evaluation Criteria To evaluate the projects intended impact, reviewers will consider the extent to which: 1. The condition(s) identified in the problem statement will be addressed 2. The lives of community members and beneficiaries will improve To evaluate the impact indicator(s), reviewers will consider the extent to which: How impact will be measured with at least one impact indicator using the same measure at three points in time, baseline (beginning of project), end of project, and three years postproject.

Reviewers read application & evaluate using FOA criteria Yes No Reviewers write comments &

Reviewers read application & evaluate using FOA criteria Yes No Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM PSR Approval Is the PSR factually accurate? Does the PSR adhere to the FOA Evaluation Criteria? Is the PSR Consistent? Do the comments justify the score? Is the PSR polished and well-written? Approved!

Roles and Responsibilities Recap Reviewer 1. Evaluate the Application 2. Document evaluation by scoring

Roles and Responsibilities Recap Reviewer 1. Evaluate the Application 2. Document evaluation by scoring and justifying score with comments 3. Submit score and comments to facilitator through ARM 4. 5. 6. 7. Facilitator Create initial PSR by compiling all reviewer comments Holds panel meeting with reviewers Revises PSR based on reviewer comments, discussion or SAM/PAM recommendations Submit PSR to SAM through ARM SAM/PAM Responsibility 8. Determine whether the PSR can be approved. If not, back to step 6. Facilitator Responsibility 9. Share final PSR with reviewers to ensure it reflects panel opinion. SAM Responsibility 10. Submit PSR to PAM for approval through ARM PAM Responsibility 11. Approve the PSR