Design Science Research Rigorous and Relevant Alan R

Design Science Research: Rigorous and Relevant Alan R. Hevner University of South Florida National Science Foundation

NSF Disclaimer l Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation. November 2007 Arizona Seminars 2

Outline Introduction l Design Science Research l Research Impacts l Design Research Guidelines l l Three Cycles of Design Research Relevance l Rigor l Design – Build and Evaluate l l Questions and Discussion November 2007 Arizona Seminars 3

Research Portfolio l l l Ph. D. in Computer Science from Purdue Faculty Member at Minnesota (CS), Maryland (IS), and USF (IS) Database Systems l l l Software Engineering l l l Query Optimization on Distributed Database Systems Query and File Allocation Algorithms Cleanroom Software Engineering Metrics and Software Testing Information Systems Analysis and Design l l Health Care Data Warehousing and Data Mining Telecommunication Systems November 2007 Arizona Seminars 4

Design Science Research l Sciences of the Artificial, 3 rd Ed. – Simon 1996 l l l Design in Other Fields – Long Histories l l l A Problem Solving Paradigm The Creation of Innovative Artifacts to Solve Real Problems Engineering, Architecture, Art Role of Creativity in Design Research in Information Systems l l How to Perform Research in Design Science ! Formal Design Science Research Theories ? November 2007 Arizona Seminars 5

MISQ 2004 Research Essay A. Hevner, S. March, J. Park, and S. Ram, “Design Science Research in Information Systems, ” Management Information Systems Quarterly, Vol. 28, No. 1, March 2004, pp. 75 -105. Historically, the IS field has been confused about the role of ‘technical’ research. Technical researchers felt out of the mainstream of ICIS/MISQ community. l l Formation of Workshop on Information Technology and Systems (WITS) in 1991 Initial Discussions and Papers l l l Iivari 1991 – Schools of IS Development Nunamaker et al. 1991 – Electronic GDSS Walls, Widmeyer, and El Sawy 1992 – EIS Design Theory March and Smith 1995 from WITS 1992 Keynote Encouragement from IS Leaders such as Gordon Davis, Ron Weber, and Bob Zmud Allen Lee, EIC of MISQ, invited authors to submit essay on Design Science Research in 1998 l l l Four Review Cycles with multiple reviewers Published in 2004 November 2007 Arizona Seminars 6

MISQ Paper Impacts l Professional Impact l l Citation Impact l l Raised visibility of DSR in IS Identified interdisciplinary synergies (e. g. , CS, Engineering design, management, etc. ) Identified relationships among research paradigms (e. g. , behavioral, economic, etc. ) Over 200 citations on Google Scholar International Impact Doctoral Education and Research Impact Concerns: l l l Over reliance on seven guidelines for research design Silence on role of theory in DSR Pragmatic nature of DSR l November 2007 Roles of Rigor and Relevance Arizona Seminars 7

DESRIST Conferences l Design Science Research in Information Technology and Systems Claremont 2006 l Pasadena 2007 l Atlanta 2008 – May 7 -9, 2008 l Interdisciplinary Participation l Website: l l http: //desrist 2008. cis. gsu. edu/ November 2007 Arizona Seminars 8

European Conference on IS June 7 -9, 2007 l l l St. Gallen, Switzerland ECIS Theme - “Relevant Rigour – Rigorous Relevance” Keynote Address l l Panel l l Title – “Design Science Research as a Rigorous Approach for Relevant Solutions” “Is Design Science Research the Answer to the Relevance Problem? ” AACSB Call for Relevant Business Research in “Impact of Research” Report November 2007 Arizona Seminars 9

International Conference on IS December 9 -12, 2007 l l Montreal, Canada Design Science Track l l l 40 Submissions – 10 Papers Accepted At ICIS 2006 in Milwaukee, 52 Submissions – 12 Papers Accepted Workshop on Information Technology and Systems (WITS) November 2007 Arizona Seminars 10

Design Science Research in Major IS Journals l MISQ Special Issue on Design Science to appear in 2008 l New Senior Editor and Associate Editors for DS Papers l l ISR l Encouragement of DS Submissions Role of ACM and IEEE Publications l Senior Scholars List of Top IS Journals l November 2007 Arizona Seminars 11

IS Research Framework Information Systems (IS) are complex, artificial, and purposefully designed. l IS are composed of people, structures, technologies, and work systems. l Two Basic IS Research Paradigms l Behavioral Research – Goal is Truth l Design Research – Goal is Utility l November 2007 Arizona Seminars 12

IS Research Cycle November 2007 Arizona Seminars 13

Design Science l Design is a Artifact (Noun) l l l Design is a Process (Verb) l l l Constructs Models Methods Instantiations Build Evaluate Design is a Wicked Problem l l l Unstable Requirements and Constraints Complex Interactions among Subcomponents of Problem and resulting Subcomponents of Solution Inherent Flexibility to Change Artifacts and Processes Dependence on Human Cognitive Abilities - Creativity Dependence on Human Social Abilities - Teamwork November 2007 Arizona Seminars 14

Environment IS Research Relevance Rigor People • Roles • Capabilities • Characteristics • Experience Organizations • Strategies • Structure • Culture • Processes Foundations Develop / Build • Theories • Artifacts Applicable Knowledge Business Needs Assess Technology • Infrastructure • Applications • Communications Architecture • Development Capabilities Application in the Appropriate Environment Knowledge Base Refine • Theories • Frameworks • Experimental Instruments • Constructs • Models • Methods • Instantiations Methodologies Justify / Evaluate • Analytical • Case Study • Experimental • Field Study • Simulation • Experimentation • Data Analysis Techniques • Formalisms • Measures • Validation Criteria • Optimization Additions to the Knowledge Base

Guidelines for DS Research in IS Purpose of Seven Guidelines is to Assist Researchers, Reviewers, Editors, and Readers to Understand Evaluate Effective Design Science Research in IS. l Researchers will use their creative skill and judgment to determine when, where, and how to apply the guidelines to projects. l All Guidelines should be addressed in the Research. l November 2007 Arizona Seminars 16

Design Science Guidelines Guideline Description Guideline 1: Design as an Artifact Design-science research must produce a viable artifact in the form of a construct, a model, a method, or an instantiation. Guideline 2: Problem Relevance The objective of design-science research is to develop technology-based solutions to important and relevant business problems. Guideline 3: Design Evaluation The utility, quality, and efficacy of a design artifact must be rigorously demonstrated via well-executed evaluation methods. Guideline 4: Research Contributions Effective design-science research must provide clear and verifiable contributions in the areas of the design artifact, design foundations, and/or design methodologies. Guideline 5: Research Rigor Design-science research relies upon the application of rigorous methods in both the construction and evaluation of the design artifact. Guideline 6: Design as a Search Process The search for an effective artifact requires utilizing available means to reach desired ends while satisfying laws in the problem environment. Guideline 7: Communication of Research Design-science research must be presented effectively both to technology-oriented as well as management -oriented audiences.

Design Science Case Studies l Three Exemplars in MISQ Paper l l Gavish and Gerdes DSS 1998 Aalst and Kumar ISR 2003 Markus, Majchrzak, and Gasser MISQ 2002 Recent Doctoral Research Project l l Monica Tremblay - Uncertainty in the Information Supply Chain: Integrating Multiple Health Care Data Sources Artifacts: l l ISC Metrics – Completeness, Volatility User Presentations of Metrics Evaluation – Focus Groups Details November 2007 Arizona Seminars 18

Scandinavian Journal of IS l Upcoming Issue on Design Research Iivari, “A Paradigmatic Analysis of Information Systems as a Design Science” l Hevner, “A Three Cycle View of Design Science Research” l Other Responses to the Iivari paper l November 2007 Arizona Seminars 19

Three Cycles of DS Research Environment Design Science Foundations Application Domain • People • Organizational Systems • Technical Systems Relevance Cycle • Requirements Build Design Artifacts & Processes November 2007 • Scientific Theories & Methods • Experience Rigor Cycle Design Cycle & Expertise • Grounding • Additions to KB • Field Testing • Problems & Opportunities Knowledge Base Evaluate Arizona Seminars • Meta-Artifacts (Design Products & Design Processes) 20

The Relevance Cycle l The Application Domain initiates Design Research with: l l l Field Testing of Research Results l l Research requirements (e. g. , opportunity, problem, potentiality) Acceptance criteria for evaluation of design artifact in application domain Does the design artifact improve the environment? How is the improvement measured? Field testing methods might include Action Research or Controlled Experiments in actual environments. Iterate Relevance Cycle as needed l l l Artifact has deficiencies in behaviors or qualities Restatement of research requirements Feedback into research from field testing evaluation November 2007 Arizona Seminars 21

The Rigor Cycle l Design Research Knowledge Base l l l Design Theories Engineering Methods Experiences and Expertise Existing Design Artifacts and Processes Research Rigor is predicated on the researcher’s skilled selection and application of appropriate theories and methods for constructing and evaluating the artifact. Additions to the Knowledge Base: l l l Extensions to theories and methods New experiences and expertise New artifacts and processes November 2007 Arizona Seminars 22

Design Theories Is a kernel ‘Design Theory’ essential for rigorous design research? l I would contend that the answer is No. l Design research can be grounded on: l Design Theory l Opportunities, Problems, Potentialities l Analogies, Metaphors l Creative Inspiration and Insight l November 2007 Arizona Seminars 23

Design Cycle l Rapid iteration of Build and Evaluate activities l l The hard work of design research Build – Create and Refine artifact design as both product (noun) and process (verb) Evaluation – Rigorous, scientific study of artifact in laboratory or controlled environment Continue Design Cycle until: l l Artifact ready for field test in Application Environment New knowledge ready for inclusion in Knowledge Base November 2007 Arizona Seminars 24

Design Science Challenges l l l Inadequate Theory Base for Scientific and Engineering discipline Insufficient Sets of Constructs, Models, Methods, and Tools in Knowledge Base to Represent real-world Problems and Solutions Design is still a Craft relying on Intuition, Experience, and Trial-and-Error Design Science Research is perishable as technology advances rapidly Rigorous Evaluation Methods are difficult to apply in Design Science Research Communication of Design Science Results to Managers is Essential but a Major Challenge November 2007 Arizona Seminars 25

Questions and Discussion November 2007 Arizona Seminars 26

Design as an Artifact l l l The IT Artifact is the ‘core subject matter’ of the IS field. Artifacts are innovations that define the ideas, practices, technical capabilities, and products through which the analysis, design, implementation, and use of IS can be accomplished. Design Science Research in IS must produce an Artifact l l Construct, Model, Method, Instantiation Research Design vs. Routine Design l Innovation vs. Use of Known Techniques November 2007 Arizona Seminars 27

Problem Relevance Research Motivation l The Problem must be real and interesting. l Problem solving is a search process using actions to reduce or eliminate the differences between the current state and a goal state [Simon 1999]. l Design Science Artifact must be relevant and useful to IS practitioners - Utility. l November 2007 Arizona Seminars 28

Design Evaluation Rigorous Evaluation of the Utility, Quality, and Beauty (i. e. , Style) of the Design Artifact. l Evaluation provides feedback to the Construction phase for improving the artifact. l Design Evaluation Methods l November 2007 Arizona Seminars 29

Design Evaluation Methods 1. Observational Case Study – Study artifact in depth in business environment Field Study – Monitor use of artifact in multiple projects 2. Analytical Static Analysis – Examine structure of artifact for static qualities (e. g. , complexity) Architecture Analysis – Study fit of artifact into technical IS architecture Optimization – Demonstrate inherent optimal properties of artifact or provide optimality bounds on artifact behavior Dynamic Analysis – Study artifact in use for dynamic qualities (e. g. , performance) 3. Experimental Controlled Experiment – Study artifact in controlled environment for qualities (e. g. , usability) Simulation – Execute artifact with artificial data 4. Testing Functional (Black Box) Testing – Execute artifact interfaces to discover failures and identify defects Structural (White Box) Testing – Perform coverage testing of some metric (e. g. , execution paths) in the artifact implementation 5. Descriptive Informed Argument – Use information from the knowledge base (e. g. , relevant research) to build a convincing argument for the artifact’s utility Scenarios – Construct detailed scenarios around the artifact to demonstrate its utility

Research Contributions l l l What is New and Interesting? Does the Research make a clear contribution to the business environment, addressing a relevant problem? The Design Artifact l l Foundations l l Exercising the artifact in the problem domain adds value to the IS practice Extend and improve foundations in the design science knowledge base Methodologies l Creative development and use of methods and metrics November 2007 Arizona Seminars 31

Research Rigor l l Use of Rigorous Research techniques in both the Build and Evaluate phases Building an Artifact relies on mathematical foundations to describe the specified and constructed artifact. l l l Principles of Abstraction and Hierarchical Decomposition to deal with Complexity Evaluating an Artifact requires effective use of techniques in previous slide. Research must be both Relevant and Rigorous November 2007 Arizona Seminars 32

Design as a Search Process l Good design is based on iterative, heuristic search strategies. Simon’s Generate/Test Cycle l Problem Simplification and Decomposition l Modeling Means, Ends, and Laws of the Problem Environment l The Search for Optimal Solutions may not be feasible or tractable. l The Search for Satisfactory Solutions may be the best we can do - Satisficing l November 2007 Arizona Seminars 33

Communication of Research l Technical audiences need sufficient detail to construct and effectively use the artifact. l l Managerial audiences need an understanding of the importance of the problem and the novelty and utility of the artifact. l l How do I build and use the artifact to solve the problem? Should I commit the resources (staff, budget, facilities) to adopt the artifact as a solution to the problem? Research presentation must be fitted to the appropriate audience (e. g. , journal). November 2007 Arizona Seminars 34

Problem Relevance l Problem Statement: Public policy knowledge workers draw on a set of pre-existing tools when acquiring data from multiple data sources available from the information supply chain. The data acquisition process and the task of correctly combining and manipulating data from multiple data sources in the information supply chain have many challenges: data are unbounded, data definitions and schemas vary, and there is no guarantee of data quality. In most cases, knowledge workers make decisions with available information and use “gut instinct” or experience to choose the correct course of action when data sources conflict or do not match expectations. These challenges are made even more complicated by the knowledge worker’s own judgment biases. Existing tools can aid knowledge workers, yet the lack of integration among these tools aggravate cognitive and behavioral biases and result in missed opportunities for knowledge creation. November 2007 Arizona Seminars 35

Research Questions Can we design metrics and a decision method that will integrate multiple data sources with varying degrees of data quality in the IS supply chain to better support public policy decision makers? l Can we evaluate the utility, quality, and efficacy of the metrics and method? l November 2007 Arizona Seminars 36

Research Rigor l Designs will be grounded by: Data Products Foundations [Shankaranarayan et al. 2003] l Data Quality Foundations [Wang et al. 1997] l Behavioral Decision-Making Foundations [Tversky and Kahneman 1982] l l Design Evaluation will be performed in two phases: Field Study of Public Health Decision-Makers l Focus Groups of Experts l November 2007 Arizona Seminars 37

Design as a Search Process Two Research Cycle Iterations l Cycle 1: l Initial Designs and Instantiation l Practical Evaluation in Field Study with Feedback l l Cycle 2: Improved Design and Instantiation l Evaluation in Focus Groups to Study Impacts on Behavioral Biases l November 2007 Arizona Seminars 38

Design as an Artifact l The Artifacts are: Data Quality Metrics on Data Products l New Algorithms for Calculating Data Quality Metrics on Data Products l New Methods for Comparing and Integrating Data Products l New Human-Computer Interface Presentations to support Decision-Making l November 2007 Arizona Seminars 39

Design Evaluation Cycle 1 – Field Study of How Public Health Policy workers use initial metrics to support Decision-Making. l Cycle 2 – Focus Groups on the use of the metrics and methods to study User Biases on Public Health Policy Decision-Making l Experts in the Health Care Field l Scenarios will be drawn from Field Study l November 2007 Arizona Seminars 40

Research Contributions l The Design Artifacts l l Metrics, algorithms, methods and interfaces will add clear value to Public Health Policy environments Foundations l l l New models and algorithms for calculating data quality metrics New methods of integrating multiple data products New methods of data product presentation to decisionmakers November 2007 Arizona Seminars 41

Communication of Research Presentation of Research Results in topquality IS journals and conferences l Initial study that motivated this project: l l Tremblay, Fuller, Berndt, and Studnicki, “Doing More with More Information: Changing Healthcare Planning with OLAP Tools, ” Decision Support Systems, In Press, Available on DSS Elsevier Website, 2006. November 2007 Arizona Seminars 42