2019 XSEDE Staff Climate Study Lizanne De Stefano

2019 XSEDE Staff Climate Study Lizanne De. Stefano, Lorna Rivera, Julie Wernert XSEDE External Evaluation Team

Purpose Support XSEDE’s organizational health by providing data over time to: • Better understand current working conditions • Recognize successes and areas of concern • Develop responses to improve working conditions • Improve workplace efficiency and satisfaction 2

Method • Annual on-line Survey to all XSEDE staff and leadership • Core items • XSEDE specific items • May-June administration • Disaggregation by Level 2, 3, site, FTE, length of employment • Special requests by L 2 and L 3 managers • EXTENSIVE dissemination and interaction around results • Documentation of XSEDE response to results • Quarterly Meetings 3

Study Response Rates 100% 75% 50% 62% 49% 65% 56% 52% 56% 57% 2017 (N=181) 2018 (N=206) 25% 0% 2013 (N=230) 4 2014 (N=264) 2015 (N=298) 2016 (N=249) 2019 (N=189)

Respondent Demographics, N=115 Staff Level (Self Report) Length of Employment (Self Report) 100% Gender (Self Report) 71% 69% 57% 18% 2% 9% 22% 3% 3% 0% 0% Other 5 Level 1 Level 3 or 2 manager 28% Staff 0% <1 year 1 -3 3 -5 5+ years Other Female Male

2019 General Findings • All but one (Wiki & Website) index increased in ratings from 2018 to 2019, however none of these differences were found to be statistically significant. The highest gains were made in Communication Tools and Support & Belonging. • Five of the eight study dimensions achieved their highest ratings to date. These include Communication Tools, Decision Making, Leadership & Management, Support & Belonging, and Value & Satisfaction. • Staff were more satisfied with the progress their program area is making towards their goals (p=. 032) and increasingly believe some of their best work has been done on XSEDE (p=. 021). • All groups rated Equity positively (above a 3. 7), though some Gender and L 2 differences exist. Racial and ethnic URMs, however, rated the index similarly to majority groups. • Comments suggest a lack of awareness regarding staff training resources for internal XSEDE communications and management (i. e. Jira, Confluence Wiki, and performance evaluations). 6

3, 87 3, 81 3, 94 4, 02 [VALUE]*** 3, 81 3, 90 [VALUE]*** 2019 (N=115) 4, 08 4, 16 [VALUE]*** 2018 (N=117) 4, 14 4, 16 [VALUE]* 4, 12 4, 18 3, 66 3, 73 4 [VALUE]*** 2013 (N=124) 4, 26 4, 36 5 [VALUE]** 2013 (baseline), 2018, and 2019 Index Scores (Scale 1 – 5) 3 2 1 Communication Tools * p <. 05, ** p <. 01, *** p <. 001 7 Decision Making Equity Inclusion Leadership & Management Support & Belonging Value & Satisfaction Wiki & Website

2019 Recommendations • Training & Human Resources: Improve awareness regarding optional staff evaluations and training on project management tools. Consider coupling implementation of any new processes with training available to all XSEDE staff. Identify and advertise high quality management training resources for interested staff. • Unity: Reduce staff isolation by offering more opportunities to engage with XSEDE colleagues. Consider piloting optional programs like “Catch Up with a Colleague” where staff can informally engage with co-workers they may not normally get to interact with. Promote and reward cross-area collaborations with project-wide recognition and support for dissemination activities (i. e. conference presentations, seminars, training workshops, etc. ). 8

2019 XSEDE User Satisfaction Survey Lizanne De. Stefano, Lorna Rivera, Julie Wernert XSEDE External Evaluation Team

Purpose • Designed as a basic report card for leadership, service-area owners, and project managers - a tool to gauge overall awareness and satisfaction of a wide range of services among a broad range of users • Other assessments and feedback mechanisms are deployed throughout the Project to augment these results 10

Method • Broad input from project stakeholders • Engaged each project area, leadership, project managers, User Advisory Board, etc. to refine/update questionnaire • Resulted in several new items added for evaluation; others were retired or refined • Items evaluated align closely with project metrics and reporting requirements • Broad Sampling • Sample of 5000 users derived from those who interacted with XSEDE resources in preceding 24 months • Oversampled to include a greater proportion of those in traditionally underrepresented groups • Disaggregation by • User type (i. e. , faculty, graduate students, post-docs, etc. ) • Institution type (i. e. , MSIs, EPSCo. R state institutions, Associate’s-degree-granting colleges, doctoralgranting institutions, etc. ) • Broad dissemination to all areas of the project, as well as users, Campus Champions, etc. ; results inform several project-level KPIs • Results are made available at the June quarterly meeting 11

Response Rate • 19% final response rate (925 cases) - after recodes • Down from 22% rate of response (1107 cases) in 2018 • Response peaked in 2015 27. 4% • Sample size and representation are good • Factors that may influence response rate • Stakeholder survey conducted by Gershenfeld, Berente, et al fielded during the same timeframe to overlapping populations, potentially causing confusion and frustration • i. e. , Didn’t I already to this? , How is this different that the survey I already took? • Employed fewer reminders due to complaints received during the 2018 administration • Survey fatigue – users anecdotally (and in text comments) report being over-surveyed • Project age - as project ages, some may be less inclined to participate • Drop in response correlates with national trends for surveys without completion incentives • But a higher response rate does not necessarily mean better/more valid data 12

2019 Population Overview • Respondent profile is similar to that of previous years • 34% faculty, 28% graduate students, 16% post-docs, and 14% university research staff • 6% indicated they were a Campus Champion • Of respondents indicating they had used XSEDE resources, 31% had been using XSEDE resources for less than one year, 27% for one to two years, and 38% for three or more years • Distribution nearly identical to prior year • The consistent proportion of new users may explain the consistently lower awareness rankings that lag behind satisfaction rankings • Over 85% of respondents consider themselves to be at least somewhat experienced XSEDE users, ranking themselves at 3 or higher on a 5 -point scale (where 1 is “not at all experienced” and 5 is “extremely experienced”) • Up from 76% in 2018 • The most typical XSEDE user is likely to be a white/non-Hispanic, male, working at a researchfocused/doctoral-granting institution in the physics, biology, chemistry, engineering or computer science field 13

2019 General Findings • Aggregate awareness of XSEDE resources and services: 3. 73/5. 00 • Up from 3. 69 in 2018 • Overall satisfaction with XSEDE services remains high: 4. 35/5. 00 • As has been the case since the beginning of XSEDE 1, all areas evaluated were significantly above 3. 0 (on a 5. 0 scale) • While satisfaction generally trended upward, changes in satisfaction (increases and decreases) were modest, indicating the project is mature, stable, and consistently delivering a broad range of services at a high level • Importance of XSEDE remains high at 4. 22/5. 00 • 49% of respondents report the use of XSEDE-associated or -operated resources are essential in conducting their research program 14

Importance of XSEDE • Importance of XSEDE remains high, but dipped from its historical high of 4. 43 • Not surprising given where we are in the life of the project – as the project moves closer to its scheduled end date, users begin to factor in other resources, look for alternatives with a longer life, seek more certainty 15

2019 Recommendations • Increase computational resources to the national community • Expand software and data storage capabilities • Augment training opportunities, especially for new users • Offer a more flexible and dynamic allocations process that adapts to users’ needs • Provide more guidance/feedback to PIs of rejected allocations proposals • Increase/expand outreach activities to users from Minority-serving, teachingfocused and EPSCo. R state institutions, as well as to members of traditionally underrepresented groups 16