Using Veera with R and Shiny to Build

Using Veera with R and Shiny to Build Complex Visualizations Bharathwaj Vijayakumar, Senior Business Intelligence Analyst, Rowan University #RIConf 2019

Overview of the presentation • Using Veera with R and Shiny to Build Complex Visualizations • Documentation and Veera Workflow

Objectives • Visualize the number of students migrating from one major to another • Visualize the sequence of courses that students take in a specific program • Visualize the effect of grades in one course, on the courses that follow that

Tools and Technology Used • SQL • Veera • R • Shiny R and Shiny was used because Tableau did not directly support the chart types needed for this type of data

Visualizations Used and Why? Sankey Chart Advantages • Look at magnitude of a flow from point A to point B • Very easy to interpret and visually pleasing • Can be created as multilevel, A to B to C Disadvantages • Not Excel or table! (Can be an advantage too!) • Sometimes flows can looks similar • Scale can be hard to interpret sometimes • Needs dataset to be in a specific way • Cannot Handle cyclic flows (A to B to A)

Visualizations Used and Why? Chord Diagram Advantages • Look at magnitude of a flow from point A to point B • Visually pleasing • Can handle cyclic bidirectional flows A to B to A Disadvantages • Not Excel or table! (Can be an advantage too!) • Sometimes flows can looks similar • Needs dataset to be in a specific way • Can be difficult to interpret compared to Sankey

Visualizations Used and Why? • Sankey can be used to show the flow of students from one major to another • Chord Diagram can help to show the sequence of courses that students take and grades they get. • Since students can take courses out of order, there can be bidirectional flows. Chord diagram works better than Sankey for course data

Overall Workflow Choose the type of visualization Prepare the dataset using Veera Write Code in R Write the necessary code for Shiny Publish to Shiny Server

Objectives • Visualize the number of students migrating from one major to another

Data Preparation in Veera Migration between Majors Current Format • One row per student per term enrolled • It has major, ids, college and other details Required Format Source Target Flow Source Major (Original Major of students) Target Major (Majors they transferred into) Number of Students

Data Preparation in Veera Migration between Majors Final Format with years

Data Preparation in Veera - Migration between Majors Fall 2010 Counts the number of students Fall 2011

Data Preparation in Veera - Migration between Majors Job called multiple time for different years Calculates the diff bet the first term and the enrolled term to find the year Writes to two different outputs

Visualizations in R Shiny - Migration between Majors • All Years - https: //rowan. shinyapps. io/migration_between_majorsall_years/ • All Years by Source College https: //rowan. shinyapps. io/migration_between_majorsall_years_by_source_college/ • Each Year by Source College https: //rowan. shinyapps. io/migration_between_majorseach_year__by_source_college/

Objectives • Visualize the sequence of courses that students take in a specific program • Visualize the effect of grades in one course, on the courses that follow that

Data Preparation in Veera Course Sequence Current Format • One row per student per term per course registered • It has major, course, grades, ids, college and other details Required Format Source Target Flow Source Course Target Course Number of Students

Data Preparation in Veera – Course Sequence Rest of the data cleanup

Data Preparation in Veera Course Sequence Final Format

Visualizations in R Shiny – Course Sequence Chord Diagram https: //rowan. shinyapps. io/Major_Pathways_Chord_Diagram/

Documentation & Workflow in Veera

Documentation & Workflow in Veera Setup the users and connections Development in Sandbox Folder Create Documentation Prod &Test and share with user and schedule Move the job to QA folder and share for QA Moveup folder share with Admin Users

Documentation & Workflow in Veera Setup the users and connections • Admin sets up the connections and the folders. • Windows active directory accounts are not used • All are veera based accounts maintained by admins Connections are setup using service accounts

Documentation & Workflow in Veera Development • Always use shared drives for input excel or text files and use full path. Create separate input and output folders for Veera • Use Query nodes and put in SQL with a basic select *, rather than importing tables Save the job under sandbox folder. Outputs • The output of test jobs should always write to Test schema using the service account • They should all have _test in the file names or table names • Create veera outputs under shared drive by creating appropriate folders

Documentation & Workflow in Veera Documentation • Add numbers to your nodes by right clicking and renaming it. This helps to debug the steps when a job fails. • Add notes to each node by right clicking it

Documentation & Workflow in Veera Documentation • Source control veera job using Github by exporting the jobs

Documentation & Workflow in Veera Documentation • Document everything in a web page or a link • Each step number in the veera job will have the details in the documentation page.

Documentation & Workflow in Veera Documentation • When you are inside a job, click on job-edit notes and add a detailed note about the job and its documentation link

Documentation & Workflow in Veera QA and Prod Moveup • • Once the job is done, copy it to QA folder and share it with some other user to review it Once QA is done, copy the job to the Moveup folder and share it with admin Admin will check the outputs, make sure it is changed to prod and then schedule the jobs. The admin will share the jobs in Prod and Test folder with the user, for running or debugging in the future Test jobs will be properly labeled by admins

Includes only First Time UG cohorts Pulls in courses and grades for each term ID = 12345 Good Grade Fall 2010 Spring 2011 Math 2 Chem 1 Chem 2 Pulls in courses and grades taken after the current course in the terms greater than the current term

Thank you! Questions/Comments? Bharathwaj Vijayakumar, Senior Business Intelligence Analyst, Rowan University vijayakumar@rowan. edu #RIConf 2019