NASA Goddard Space Flight Center Systems Engineering Seminar

NASA Goddard Space Flight Center Systems Engineering Seminar March 3, 2009 Shared Knowledge and Information Flow in Systems Engineering Socio-Cognitive Analysis of the GSFC Mission Design Laboratory Mark S. Avnet Ph. D. Candidate Engineering Systems Division Massachusetts Institute of Technology

Who Am I and Why Am I Here? • S. B. in Physics, MIT, 2001; M. A. in Space Policy, GWU, 2005 • Software Engineer, 2001 – 2003; NASA HQ, 2004 – 2005 • Observed an Interesting Phenomenon ― Decisions in space systems development require integration of perspectives: policy, scientific, engineering, public, etc. ― Systems engineering takes into account the unique views of each, but the engineer is taken to be outside of the stakeholder framework. • Ph. D. in Engineering Systems, MIT, 2009 ― Research addressing this issue ― Focus of this talk: contributions to SE here at GSFC Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 2 of 37

Perspectives on Space Systems Design Source: Robinson, G. L. , “Systems Engineering Initiatives at NASA, ” Goddard/SMA-D Education Series, 25 Sept 2008. Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 3 of 37

Structure of the Presentation Overview of the Mission Design Lab 1 2 Analysis of the Design Process A Model of Shared Knowledge 3 4 Integrated Analysis: People and Process Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 4 of 37

Part 1 Overview of the Mission Design Lab Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 5 of 37

GSFC Integrated Design Center Focus of this Talk Integrated Design Center (IDC) Mission Design Lab (MDL) Instrument Design Lab (IDL) Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 6 of 37

The Mission Design Lab Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 7 of 37

The MDL: Structure and Products http: //idc. nasa. gov/mdl/products. cfm http: //idc. nasa. gov/idc/services. cfm Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 8 of 37

The MDL: Roles and Facility Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 9 of 37

MDL Design Study Observations “Typical” Studies Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 10 of 37

Part 2 Analysis of the Design Process Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 11 of 37

The Design Structure Matrix (DSM) Task A depends on information from Task G Tasks D and E must be done concurrently Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 12 of 37

Design Process Analysis Series Coupled Phases of the Design Life Cycle Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Starting Assumptions Parallel Slide 13 of 37

Modeling the MDL Design Process Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 14 of 37

Partitioning the DSM: The Conceptual Design Lifecycle Requirements Definition Phase Engineering Design Phase Integration Phase Maintenance and Support Phase Costing Phase Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 15 of 37

Critical Design Trades and Interdependent Disciplines 13 Core Loop Types Spacecraft Bus Loop Propulsion Sizing Loop Stabilization Loop Ground Segment Loop Data Loop Power System Electronics Loop Power Loop Electrical Heating Loop Propulsion Thermal Control Loop Radiator Operation Loop Reentry Loop Computing Reliability Loop Radiation Shielding Loop Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 16 of 37

Tearing the DSM: Indentification of Starting Assumptions Tear the Design Budgets Power Budget Mass Budget Reliability Budget Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 17 of 37

The Torn DSM: MDL Process with Starting Assumptions Made Requirements and Assumptions Phase Orbit Determination Phase Sequential Engineering Design Phases e e It t a r Integration Phase Costing Phase Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 18 of 37

The Core of Interdependent Disciplines Flight Dynamics Location Mission Operations Avionics Location Communications Electrical Power Location Mechanical Thermal Data Loop Ground Segment Loop Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 19 of 37

Insights from DSM-Based Analysis The Design Structure Matrix is a powerful tool for describing and analyzing the space systems design process. (Results for your system may vary. ) Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 20 of 37

Part 3 A Model of Shared Knowledge Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 21 of 37

Mental Models of the System Mental Models “Mechanisms whereby humans are able to generate descriptions of system purpose and form, explanations of system functioning and observed system states, and predictions of future system states”* * Rouse, W. B. and N. M. Morris (1986). “On Looking Into the Black Box: Prospects and Limits in the Search for Mental Models. ” Psychological Bulletin 100(3): 349– 363. Shared Mental Model (SMM) Condition in which two people utilize the same underlying mechanisms or at least utilize mechanisms that lead to similar descriptions, explanations, and predictions Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 SMM Team Member Slide 22 of 37

Measuring Mental Models Survey Question on Major Design Drivers Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 23 of 37

Measuring Shared Mental Models Mental Model Sharedness, Sx, y , is defined as: Ratio of common choices to total choices Dx = # of drivers selected by person x Dy = # of drivers selected by person y Dx, y = # of drivers selected by both x and y Sx, y Team Member x Team Member y Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 24 of 37

Social Network Analysis A set of tools and techniques for analyzing a large group of entities (nodes) and the structure of interactions and/or relationships among them (edges). Node Edge Node = Design Team Member x or y Edge = Shared Mental Model between x and y Edgeweight = Value of Sharedness, Sx, y Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 25 of 37

Dynamics of Shared Knowledge Post-Session Pre-Session CSMM = structural similarity (edge-by-edge correlation) Change in Shared Knowledge Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 26 of 37

Dynamics of Shared Knowledge: Relationship to System Attributes Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 27 of 37

Part 4 Integrated Analysis: People and Process Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 28 of 37

Content of Shared Knowledge: Perceived Importance of Drivers IP, Comm = proportion of team checking Communications Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 29 of 37

The Communications Subsystem: An Indicator of Shared Knowledge Recall the Central Role of Communications in the Design Process Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 30 of 37

Measuring Team Coordination Expected Interaction Matrix Based on Core Loop Types in the Partitioned DSM Actual Interaction Matrix Based on Survey Data of Interactions for Each Study (Study 3 Shown Here) Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 31 of 37

Socio-Technical Congruence Matrix Overlay of Expected and Actual Interactions N# = number of # cells Nb = number of blank cells N = total number of cells Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 32 of 37

Dynamics of Shared Knowledge: Relationship to Team Coordination Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 33 of 37

The Typical MDL Process: Recommendations in Discussion People Process Tools Period of learning and consensus building Resolve orbit determination trades DSM-based process automation software Determine starting assumptions Facility Design sequentially… then iterate Lab layout based on interdependent disciplines Sub-teams based on interdependent disciplines Proposed Standard Design Process Model under Development in Conjunction with the MDL Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 34 of 37

Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 35 of 37

The People Behind This Work Annalisa Weigel, MIT, Thesis Advisor NASA Graduate Student Researchers Program (GSRP) Deborah Amato, Former IDC Systems Engineer Jennifer Bracken, IDC Systems Engineer Tammy Brown, IDL Team Lead Bruce Campbell, IDC Manager Anel Flores, MDL Systems Engineer Gabriel Karpati, Former IDC Systems Engineer John Martin, MDL Team Lead Mark Steiner, SESAC Branch Head IDC Support Staff: Felicia Buchanan-Jones, Dawn Daelemans, Elfrieda Harris, Erica Robinson, Ed Young 12 MDL Customer Teams And, of course, the MDL engineers, whose sustained participation made this work possible. Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 36 of 37

Thank You Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 37 of 37

Backup Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 38 of 37

Building the DSM for the MDL Although collocation accelerates the pace of design activity, it also presents an obstacle to formal analysis and process improvement. DSM construction must account for this. • Parameter-Based DSM • Steps of DSM Construction in the MDL 1) 2) 3) 4) Preliminary Interviews Surveys on Design Sessions Structured Interviews Verification and Validation • Guiding Principles for DSM Construction in the MDL – Document maximal flow for a typical design session – Include only deliberate and purposeful information flow – Abstract two-way negotiation-type interactions Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 39 of 37

Data Collection on Mental Models • Survey Data on Major Design Drivers ― Team members indicate whether each of a set of issues drives the ultimate design. • Simple Example with Only Four Possible Drivers ― ― Cost Schedule Performance Science 24 = 16 Possible Mental Models Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 40 of 37

Filtering Out Random Responses: A Cutoff For Shared Mental Models SMMx, y = 0 35 Possible SMMs SMMx, y ≥ 1 x and y do not share mental models to any greater extent than two people with no prior knowledge of the task answering at random Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 41 of 37

Quantifying Shared Knowledge: Edge Weights in a Social Network. • Surveys Distributed: 20 Drivers and 1, 771 Possible SMMs • Network Edge Weights on a 1 -4 Scale • Time Dependence of Shared Knowledge ― 12 Design Sessions Observed ― Pre- and Post-Session Data Collected for Each Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 42 of 37

Dynamics of Shared Knowledge: Relationship to System Attributes Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 43 of 37

Propulsion Subsystem and Mission Type Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 44 of 37

Team Coordination and Shared Knowledge in the Team Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 45 of 37

Proposed Standard Design Process Model Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 46 of 37

Contributions to the Research • Product Development ― Guiding Principles for Building a Design Structure Matrix in a Rapid Collaborative Design Environment ― Method for Converting a Parameter- to a Team-Based DSM • Cross-Functional Teams and Shared Mental Models ― Scalable Network Model of Shared Knowledge in Engineering Design ― Metric that Captures Dynamics of Shared Knowledge • Systems Engineering and Space Systems Design ― System-Level Representation of the Entire Design Process ― Analysis of the Role of People in the Process ― Standardized Design Process Based on Both of the Above • Explicit Connection between Organizational/Social Psychology and Systems Engineering Best Practices Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 47 of 37

Future Work 1) Apply Methods to the Instrument Design Laboratory and to Other Similar Design Centers; Apply Both DSM and SMM Work to Longer Development Programs 2) Build DSM with Types and Strengths of Dependencies 3) Time Series Analysis – 1 to 2 Surveys Each Day Tracking the Evolution of SMMs Over Time 4) Measure SMMs Based on Other Forms of Knowledge in Addition to Task – Team, Process, Context, Competence 5) Network Analysis of Design Sessions 6) Experimental Approach with a Learning Period Structured in Various Ways and Several Combinations of Number and Length of Design Iterations Mark S. Avnet GSFC Systems Engineering Seminar March 3, 2009 Slide 48 of 37