Panorama Release Cycle Pano Dossier Requirements Approval Development

Panorama Release Cycle Pano Dossier Requirements Approval Development Test Production Testing/Approvals Production Release Performance Testing Deployment Build Dataset(s) Build Dossier Pano Team Develop Visualization Review/Optimize Dossier Engineering Unit Test Visualization PAW Dev Integration Testing Test Deployment QC Pano Dev Content Developers Pano Test PQ/DUIT/PET/ICPI Reviewers Pano Prod Panorama Developers 6

Panorama Release Migration Panorama 3 Engineering: Dossiers are migrated to Pano Dev Production Test Development 4 Performance Testing, Integration Testing and QC: Dossiers are migrated to Pano TEST 5 Release: Dossiers are migrated to Pano Production Spotlight Panorama PAW Pano for Content Developers 6 Dossiers are migrated to PAW Production Workspaces Other Projects (as needed) 2 Workspaces Daily Development: Content Developers build Dossiers in PAW Dev 1 Discovery: Users prototype Dossiers in PAW Prod > Propose New Dossiers 7

Panorama QC / Integration Testing FY 21 Q 1 Pre-clean Release February 16 th to 17 th 2021

Agenda • Panorama Dev Site Navigation & Dashboard/Dossier Selection • Testing tools • MER Structured Datasets (MSDs) • SIMS Structured Datasets (SSDs) • DATIM • Genie • Panorama QC / Integration Testing • Assignments • Integration Testing Tickets (Analytics Repo) • Testing Results & Communication • Git. Hub Notification Settings • Testing Dates and Assignments • Integration Testing Dashboard/Dossier List • Additional QC Training Resources • Timelines 9

Panorama Dev Site Navigate to https: //dev. pepfar-panorama. org/PEPFARLanding/ Username: PEPFAR Password: 2020 Public@ccount 10

Dashboard / Dossier Selection Select Dashboards from the menu: 11

Dashboard / Dossier Search for and select your dossiers and dashboards: 12

Considerations • Multiple Dossiers for some tools: o “Global” exists for certain dashboards and dossiers o Testing Single OU and Treatment Single OU have a non site version 13

Test MER Structured Datasets (MSDs) • Click this link to access the folder with the test MSDs • Select one or both of the most recently created test datasets: PSNUx. IM for Eswatini and Mozambique • A test site-level MSD file for Eswatini is also available • Note that these are test datasets only. Please do not use them for your tools or other analysis, and delete them when the final standard MER Structured Datasets are released on Friday. • The data is real, but has not been fully QC’ed. Consistency checks are running simultaneously as we are checking Panorama Dev 14

Test Operating Units For MER Data Select Eswatini or Mozambique for MER data testing: 15

Testing MER Using DATIM (If Needed) • If the data you see in Panorama does not match the data you see in the MER Structured Dataset, check against final. datim • Log in using your regular DATIM credentials • Run a DATIM Pivot Table to check the data as it was entered 16

Test SIMS Structured Datasets (SSDs) • Click this link to access the folder with the test SSDs • Select one of the six available test datasets: • • Country for DRC or Namibia Country x IM for DRC or Namibia SIMS 4. 0 Site x IM for DRC or Namibia MER Site X IM for DRC or Namibia (for Patience Experience and SIMS-MER Integrated Analysis dossiers) • Note that these are test datasets only. Please do not use them for your tools or other analysis, and delete them when the final standard SIMS Structured Datasets are released on Friday. • The data is real, but has not been fully QC’d. Consistency checks are running simultaneously as we are checking Panorama Dev 17

Testing SIMS Using Genie (If Needed) • If the data you see in Panorama does not match the data you see in the SIMS Structured Dataset, check against Genie: • SIMS 4. 0 Site x IM • SIMS Country x IM 18

Test Financial Structured Dataset (FSD) • Click this link to access the folder with the test FSDs • Note that these are test datasets only. Please do not use them for your tools or other analysis, and delete them when the final standard SIMS Structured Datasets are released on Friday. • The data is real, but has not been fully QC’d. Consistency checks are running simultaneously as we are checking Panorama Dev 19

Panorama QC / Testing Assignments • List of assignees (testers) Git. Hub issue 186 • You will be assigned to “Integration Testing” tickets in the Analytics repo • Access to the Analytics repo is granted to ICPI and program collaborators • If you don’t currently have access to the Analytic repo, please provide your Git. Hub username and signed NDA to Pano Dev Support 20

Integration Testing Tickets (Analytics Repo) https: //github. com/pepfar-datim/Analytics/issues/3072 21

QC / Testing Results & Communication • Integration Testing tickets in the Analytics repo are opened and assigned to designated testers • Upon reopening, assignees will be notified via email of their assignments and every time there is an update in the ticket (you can also search your issues by Assignee) • Scroll to the end of the ticket (some of them contain history from previous releases) and provide feedback on the results of your testing via Comment (see next slides for what to include) 22

Git. Hub Notification Settings • In order not to miss any ticket updates, please make sure that: • You get notified of all conversations you are participating in or you are tagged on • A valid email address is specified for your notifications 23

QC / Testing Instructions • Test that the visuals have updated properly to include the refreshed quarter’s data • Test the dossiers to ensure that data is: • Internally consistent (data are displaying the same across pages for the same indicator and disaggs) • Externally consistent with previous time periods using Panorama, MSD, SSD, FSD, DATIM, Excel based tools – investigate where there are issues • Test to make sure that the functionality of: • Cover pages scroll and display data as anticipated • Filters/selectors are working as anticipated • Visualizations are displaying data as anticipated 24

If Data does not match 1. Confirm that the same filters are being used to compare the data 2. Review the disaggregates and ensure they are not summing too many or too few 3. Compare against a different source MSD/SSD/FSD, DATIM, Excel testing tools, etc. 4. For Spotlight testing, remember Military data is removed, so add the “Is Military = N” to the MSD 5. Communicate the discrepancies 25

If Data Issues Are Found…Communicate Discrepancies • If you find mismatches between Panorama and the Structured Datasets/DATIM, tag Janice Holcombe (@janholcombe) and Renee Meredith(@reneerace) in ticket and provide the following information: § § § OU selected: Chapter, Page, Visual: Indicator or disagg: Time period: Filters selected: • Summarize the specific issue found § Screenshot of incorrect data, clearly highlighting the part(s) that are incorrect § Screenshot of the correct data clearly highlighted in MSD, SSD, FSD or DATIM, and showing your pivot table filters/selections • Note: Do not post data on the ICPI/DIV Git. Hub repo 26

Panorama Data Issue Example • https: //github. com/pepfar-datim/Analytics/issues/2472 • Chapter: Treatment Summary Page: Treatment Overview OU: South Africa Indicator: TX_NET_NEW Time period: Cumulative Issue: The Target for TX_NET NEW is -3, 271, 235 it should be 1, 144, 781 therefore the visual is showing -9% achievement instead of 27%. The bar on the middle right is accurately displaying 27% achievement. 27

Panorama Data Issue Example (cont’d) 28

If Functionality Issues Are Found In Panorama… • If you find any non-data issues in Panorama (i. e. export issues, selectors not working, error messages, etc. ), please tag Janice Holcombe (@janholcombe) and Renee Meredith(@reneerace) in ticket and provide the following information: • Operating Unit you used to test • Specific chapter, page and visual where issue is happening • Description of functionality error • Browser you’re using (remember: support for IE is being discontinued and Panorama is optimized for Chrome) • Screenshot of the issue (if applicable) 29

If Structured Datasets Issues Are Found… • If you find mismatches between Panorama/DATIM and the MER or SIMS Structured Datasets, please tag Janice Holcombe (@janholcombe) and Stephanie O’Hara (@oharasl) in ticket and provide the following information: • Operating Unit you used to test • Indicator(s) and/or disagg(s) that have incorrect data • Time period(s) (FY and quarter) for which the data is incorrect • Screenshot of the data that is incorrect clearly highlighted, compared with what you would expect to see based on DATIM or Panorama • Note: This process is not intended for Structure Dataset file testing. However, if while testing Pano dashboards you happen to identify a potential MSD or SSD issue, we would still like you to communicate those as well 30

If No Issues Are Found… • Close the ticket specifying that no issues were found by noon 2/18 31

Testing Dates And Timelines • QC/Integration testing: February 16 th to 17 th • If you identify and communicate issues by 2/17, you can expect that they should be resolved by the time Pano goes live • If you identify and submit Pano issues after that date, you can expect that they should be resolved during a subsequent release or fast follow • After the 2/19/21 release, please stop using the Panorama Dev/Test environment—it will be used for other activities. Start using regular pepfar-panorama. org for your analyses • If you identify issues in Panorama production after 2/19/21, please submit them through the regular Help Desk process (Panorama Support) • 32

Questions? 33

Thank you! 34