Program Learning and Leadership Outcomes Assessment in a

Program Learning and Leadership Outcomes Assessment in a Homeland Security Online Program Alexander Siedschlag, Penn State, June 6, 2018

Content / Plan for the Workshop § Workshop Objectives § Basis: The Example of the Penn State World Campus (Online) Intercollege Master of Professional Studies in Homeland Security (i. MPS-HLS) Program § § § i. MPS-HLS Program Setup, Vision, and Mission i. MPS-HLS Curriculum i. MPS-HLS PLO Assessment Approach § Program Learning Objectives (PLOs) § § § Common Core Curriculum Unifying Educational Goals and Objectives Penn State Graduate Council Scholarly and Professional Goals Exercise 1 Program-Wide i. MPS-HLS Learning Outcomes Option-Level Self-Assessment (Example) § AY 2016 -17 PLO Assessment § § AY 2016 -17 Assessment Plan AY 2016 -17 Assessment Results: HLS 803 Assignment Scores (Spring 2017) § AY 2017 -18 PLO Assessment § § AY 2017 -18 Assessment Plan Exercise 2 Instrument: Leadership Competency Questionnaire Item AY 2017 -18 Assessment Results: Exit Survey of Graduates (Fall 2017) § PLO Assessment Effects § § § Exercise 3 PLO Assessment Effects on i. MPS-HLS Assumed Broader Value for PLO Assessment on i. MPS-HLS § Securing Workshop Outcomes Alexander Siedschlag, Ph. D June 6, 2018 2

Workshop Objectives § Explain Penn State’s approach to learning outcomes assessment in a large multidisciplinary intercollege program distributed online § Assess the experience collected in design and implementation program learning outcomes assessment for such a program, with a focus on the leadership dimension § Understand evaluate the assessment instruments developed and used, including exit surveys and objective vs. competency-based measuring of program learning outcomes § Give examples of approaches and results of other programs, as represented by workshop participants § Summarize how program assessment results have informed curriculum revision and evolution, in particular to increase leadership outcomes § Compile candidates for a best practice guideline for sector-specific learning and leadership outcomes assessment Alexander Siedschlag, Ph. D June 6, 2018 3

Mission Vision i. MPS-HLS Program Vision & Mission 2018 nts: 844 g n i Spr nt stude s: 890 e e Curr graduat r So-fa Collaboration among 7 colleges of Penn State and delivered online by Penn State World Campus Alexander Siedschlag, Ph. D June 6, 2018 4

i. MPS-HLS Curriculum § Single-degree program in homeland security, 33 credits § Rooted in strong Options that provide the program’s comprehensiveness § Common core curriculum (9 credits + Orientation) realizing the program’s unifying educational goals and objectives § Base Program and Option specific required courses (12 credits) § Electives (can be cross-option and from other programs) (9 credits) § Specialization-specific 594 capstone courses (3 credits) § Optional elements of residency (summer executive short course, etc. ) Alexander Siedschlag, Ph. D June 6, 2018

i. MPS-HLS PLO Assessment Approach The Intercollege Master of Professional Studies in Homeland Security (i. MPS-HLS) program’s approach to program learning objectives/outcomes assessment is guided by: § a baseline of common core curriculum goals and specialization area learning objectives § a University-driven process, connected to regional accreditation requirements § an internal program and option review § a faculty-driven process to revisit and integrate common core curriculum and option-level outcomes into program-level PLOs § The program’s vision to prepare professionals and develop leaders for the national security workforce Alexander Siedschlag, Ph. D June 6, 2018

Common Core Curriculum Unifying Educational Goals and Objectives § Understand major policies and legislation that shapes homeland security in a globalized society § Become familiar with organizations that play a key role in the implementation of homeland security policies and administration, and recognize the interactions among them § Understand the way in which a person or group responds to a set of conditions so as to prevent and respond to incidents and catastrophic events when needed § Recognize the impact that catastrophic events, both natural and man-made, have on society and the domestic and global economy § Identify and assess potential threats, vulnerabilities, and consequences § Apply leadership skills and principles that are necessary for producing and acting on information of value within a collaborative setting § Communicate effectively in the context of particular institutional cultures § Use, conduct, and interpret research and data effectively in decision-making § Practice ethics and integrity as a foundation for analytical debate and conclusion § Develop an appreciation of the cultural, social, psychological, political, and legal aspects of terrorism and counterterrorism Alexander Siedschlag, Ph. D June 6, 2018

Penn State Graduate Council Scholarly and Professional Goals § 1. KNOW: Demonstrate appropriate breadth and depth of disciplinary knowledge, and comprehension of the major issues of their discipline § 2. APPLY/ CREATE: Use disciplinary methods and techniques to apply knowledge, and – if appropriate to the degree – create new knowledge or achieve advanced creative accomplishment § 3. COMMUNICATE: Communicate the major issues of their discipline effectively § 4. THINK: Demonstrate analytical and critical thinking within their discipline, and, where appropriate, across disciplines § 5. PROFESSIONAL PRACTICE: Know and conduct themselves in accordance with the highest ethical standards, values, and, where these are defined, the best practices of their discipline Alexander Siedschlag, Ph. D June 6, 2018

Exercise 1 § What do you see as top program learning outcomes (PLOs) for a professional master’s degree in homeland security? Alexander Siedschlag, Ph. D June 6, 2018

Program-Wide Learning Outcomes Program Learning Objectives (PLO) – Graduates from the i. MPS-HLS program will be able to… Addresses Graduate Council Goal # 1. SEARCH AND SYNTHESIZE literature to integrate homeland security principles into disciplines and professional fields. 1, 2 2. INTEGRATE the use of disciplinary methods, techniques, and knowledge to solve practical problems. 2, 4 3. IDENTIFY AND ASSESS potential threats, vulnerabilities, and consequences in 1, 4 a context from local to global environments. 4. EVALUATE scientific evidence and best practice to inform and improve realworld decisions. 2, 4 5. PRACTICE ethics and integrity as a foundation for analytical debate and conclusion. 5 6. APPLY leadership skills and principles to produce and act on information in a collaborative setting. 2, 5 7. COMMUNICATE the major issues of their discipline effectively to a diverse community of the Homeland Security Enterprise. 3, 5 Alexander Siedschlag, Ph. D June 6, 2018

Option-Level Self-Assessment (Expl. ) In table boxes, indicate whether each course or learning experience contributes to each of the program learning objectives. Indicate whether each course or other experience is designed to introduce “I”, reinforce or practice “P” or demonstrate mastery “D” of each program learning objective. Alexander Siedschlag, Ph. D June 6, 2018

So far i. MPS-HLS PLO Assessments § Mid-way through, with no in-advance notice to instructors or instructional designers § Existing assignment descriptions and grading rubrics used § AY 2016 -17 § PLO(s) measured: INTEGRATE the use of disciplinary methods, techniques, and knowledge to solve practical problems § Direct instrument: Group assignment scores in i. MPS-HLS common core curriculum course HLS 803 – Homeland Security: Social and Ethical Issues § AY 2017 -18 § PLO(s) measured: Leadership competency based on combination of two PLOs • APPLY leadership skills and principles to produce and act on information in a collaborative setting • COMMUNICATE the major issues of their discipline effectively to a diverse community of the Homeland Security Enterprise § Indirect instrument: Fall 2017 exit survey (survey of current graduates) Alexander Siedschlag, Ph. D June 6, 2018

AY 2016 -17 Assessment Plan A. PLO INTEGRATE the use of disciplinary methods, techniques, and knowledge to solve practical problems A. Student assessment title Faculty assessment of the group project assignment in common core curriculum course HLS 803 – Homeland Security: Social and Ethical Issues A. Student assessment description [shortened] Groups will form to investigate and analyze a current controversy concerning moral and political issues that involve homeland security that are connected to one of the subjects studied in the class. Students will find sources of information on the controversy (from newspapers for example), provide summaries of the main facts of the case and discuss it with other members of the group. The two case studies discussed in class can provide a model of the sort of controversy that will raise relevant issues. A. Graduate education phase Introductory phase. A. Performance criterion We expect that each group (and all students) will receive a score of 80% for the group project assignment submission. A. Implementation plan The assessment is done via grading of assignments already included in the course. Implementation will be embedded in the regular assignment grading process. Consistency of assessment will be ensured by use of a grading rubric that is included in the syllabus. A. Program review and action The results of this assessment will be reviewed by the program’s Academic Steering Committee as well as at the annual program faculty retreat. The College of the Liberal Arts in collaboration with the program chair will be responsible for acting on findings. Alexander Siedschlag, Ph. D June 6, 2018

HLS 803 Assignment Scores (Sp. 17) § Basis § All 4 Spring 2017 HLS 803 sections were included (approx. 100 students) § Results § Overall, with an average score of 93%, student achievement was excellent § The lowest individual team score received was 84%, still clearly meeting the performance criterion § Results indicate that the core curriculum very well to excellently meets the learning objective of integrating use disciplinary methods, techniques, and knowledge to solve practical problems § Interpretation and action § No significant revision needs will be identified § The College of the Liberal Arts in collaboration with the Program Chair will however review the assignment instruction and grading rubric for optimization to enable excellent student learning outcomes across teams and sections Alexander Siedschlag, Ph. D June 6, 2018

AY 2017 -18 Assessment Plan A. PLO APPLY leadership skills and principles to produce and act on information in a collaborative setting. COMMUNICATE the major issues of their discipline effectively to a diverse community of the Homeland Security Enterprise. A. Student assessment title Student self-assessment of leadership competency acquired in the program, defined by the combination of the two PLOs listed above. A. Student assessment description The program-wide i. MPS-HLS student exit survey was expanded by a question asking students to self-assess their leadership competency as acquired while in the program. The items for that question were developed based in a collection of items from relevant psychological tests and professional (HR) questionnaires. They were then pre-tested using program Academic Steering Committee, Advisory Council, and student experts. Top-rated pre-test items were then chosen for inclusion in the exit survey of graduates. The question is included in the Annex to the plan. A. Graduate education phase Demonstration of mastery phase. A. Performance criterion We expect that students’ average rating for acquired leadership competency will be above 80%. A. Implementation plan Questionnaires are sent to all i. MPS-HLS students, across the program, who applied for graduation in the Fall 2017 semester. A. Program review and action The results of this assessment will be reviewed by the program’s Academic Steering Committee as well as at the annual program faculty retreat. Alexander Siedschlag, Ph. D June 6, 2018

Exercise 2 § How would you measure leadership competency as delivered in a professional master’s degree in homeland security? § If using the questionnaire method, what items would you include to that end? Alexander Siedschlag, Ph. D June 6, 2018

Leadership Competency Q Item (1/2) § Desktop research of relevant assessment instruments § Faculty questionnaire with long list of items to rank § Narrowed down based on ranking § Short list compiled with use of Advisory Council expertise § Pre-test with a student group § One question developed for Fall 2017 i. MPS-HLS exit survey Alexander Siedschlag, Ph. D June 6, 2018

Leadership Competency Q Item (2/2) Alexander Siedschlag, Ph. D June 6, 2018

Exit Survey of Graduates (Fall 2017) § Response rate § 54% (36 out of 67) § Result § 83% of respondents assessed their leadership competency acquired through the program as either “outstanding” (54%) or “above average” (29%) § Interpretation and action § The program met its goal of providing students with leadership competency. § The program will maintain and keep updated its specific content and assignments addressing multifaceted leadership competency in homeland security, in and all-hazards and whole-community perspective. Alexander Siedschlag, Ph. D June 6, 2018 19

Exercise 3 § What larger purposes for a professional master’s degree in homeland security could PLO assessment serve? Alexander Siedschlag, Ph. D June 6, 2018

PLO Assessment Effects on i. MPS-HLS § Program Orientation revision and enforcement § HLS 801 general introductory course revision § Complete Base Program courses revision, focus on leadership competency § Review, update, and addition of assignment grading rubrics § Capstone/culminating experience revision in several Options § Program-internal evaluation of student-lifecycle process § Assessment of unity of program, and how to enhance it Alexander Siedschlag, Ph. D June 6, 2018

Assumed Broader Value for i. MPS-HLS Alexander Siedschlag, Ph. D June 6, 2018

Securing Workshop Outcomes § Best practice candidates § Value added by PLO assessment Alexander Siedschlag, Ph. D June 6, 2018 23

Contact Alexander Siedschlag, Ph. D. , M. A. Chair of Penn State Homeland Security Programs Professor of Homeland Security The Pennsylvania State University -- Penn State Harrisburg School of Public Affairs 160 W Olmsted Building 777 West Harrisburg Pike Middletown, PA 17057 Phone (717) 948 -4326 -- Fax (717) 948 -6484 http: //hbg. psu. edu/research/homelandsecurity Like i. MPS HLS on Facebook! -http: //www. facebook. com/PSU. HLS Alexander Siedschlag, Ph. D June 6, 2018