Assessing Readiness for Developmental Evaluation within the context

Assessing Readiness for Developmental Evaluation within the context of the CTSA Program Doug Easterling, Ph. D. Department of Social Sciences and Health Policy Wake Forest School of Medicine dveaster@wakehealth. edu American Evaluation Association Annual Meeting Washington, DC November 10, 2017

Organization of the Think Tank Session � Setting the context (15 mins) � Small groups (45 mins) � Overview of Developmental Evaluation and the CTSA Program How well does DE fit within CTSA? “Readiness” issues that warrant attention The 4 specific issues we’ll explore today Internal factors (within an AMC) that inhibit or facilitate DE (Clara Palfrey) Larger issues (e. g. , CTSA program requirements) that inhibit or facilitate DE (Julie Rainwater) Developing systems to enhance readiness for DE (Gaurav Dave) Measuring readiness for DE at an organizational level (Doug Easterling) Reconvene (30 mins) Report out of 3 -5 major “aha’s” and/or “points of enthusiasm” What is all this telling us about the appropriateness of DE within CTSA? Next steps ▪ To further promote the adoption of DE among CTSA hubs ▪ Conceptualizing and assessing “readiness” for DE

Developmental Evaluation � Supports the development and adaptation of innovations within dynamic environments where the problems to be addressed are complex where solutions are inherently transitory � Departure from traditional evaluation (summative and formative) Assumptions as to whether the program model should converge Dynamic interaction between findings and adaptation to design Complexity is modeled rather than trying to control the context Emergent methodology to accommodate emergent strategy Evaluator is part of innovation team rather than independent arbiter Patton, 2011 3

Institutional Context � DE raises new and ambitious expectations (e. g. , flexibility, adaptability, critical analysis, systems thinking, tolerance for ambiguity) For the evaluator For program developers For the organization that delivers the innovation � Early adopters Foundations Social service agencies Advocacy organizations Coalitions and networks 4

Academic Medical Centers � DE has the potential to advance the work of AMCs Discovery, innovation and adaptation are core elements The environment is complex and dynamic (changes in scientific knowledge, technology, financing, policy, demographics) � There are countervailing factors RCT is the dominant paradigm for research Hierarchical structures where knowledge is power Scarce resources (or at least the perception of scarcity) Admitting shortcomings can be professionally catastrophic � Not surprisingly, DE (and evaluation more generally) historically has not found a home within AMCs � Clinical and Translational Science Awards (CTSA) program offers a chance to introduce DE into AMCs (i. e. , a disruptive opportunity) 5

Clinical and Translational Science Awards (CTSA) Program � CTSA supports a national network of medical research institutions — called hubs ― that work together to improve the translational research process to get more treatments to more patients more quickly. � The hubs receive 5 -year grants from NIH to catalyze innovation in training, research tools and processes. � CTSA program is administered nationally by National Center for Accelerating Translational Sciences (NCATS) � First round of CTSA grants were awarded in 2006 � ~60 hubs are funded at any given point in time � Hubs are required to establish and maintain specific programs, and to meet specific requirements re: metrics

Sampling of Programs at a Typical CTSA Hub �Grants for pilot projects �Career development training and funding �Mentor training �Translational workforce development �Biomedical informatics �Biostatistics, Epidemiology and Research Design (BERD) consultation �Clinical Research services and facilities �Integrating special populations into research �Team science �Regulatory knowledge and support �Evaluation 7

Developmental Evaluation within the CTSA Context � In some ways, the perfect context for DE Continually increasing understanding of what it takes to promote translational science CTSA programs evolve accordingly Evaluation is a required element for each hub Opportunities for cross-program learning within each hub Remarkable sharing and co-learning across hubs � But there are countervailing factors Lack of understanding of and appreciation for DE AMCs hire scientists rather than evaluators (esp developmental evaluators) DE requires transparency and fluidity – at odds with bureaucracy NCATS emphasis on metrics and CQI – rather than exploring and adapting strategy � Some hubs are pursuing DE – either explicit DE or “looks and quacks like DE”

Conducting Developmental Evaluation within Academic Medical Centers under the Clinical and Translational Science Awards (CTSA) Program Doug Easterling, Wake Forest University (Chair And Panelist) Gaurav Dave And Tiffany Young, University Of North Carolina At Chapel Hill (Panelist) Jillian Harvey, Medical University Of South Carolina (Panelist) Janice Hogle, University Of Wisconsin-Madison (Panelist) Arthur Blank, Albert Einstein College Of Medicine (Moderator) October 27, 2016 American Evaluation Association Conference 9

What will it take to further implement DE within the CTSA context? � Increase the supply of developmental evaluators within AMCs � Increase demand for (appreciation of) a DE approach � Identify and understand barriers and facilitators (e. g. , policies, structures, attitudes, culture, leadership) at multiple levels Institution-wide at CTSA-funded medical centers The CTSA institute/center Individual leaders of specific CTSA-funded programs NCATS � Change organizational context to be more conducive to DE � Endorse DE within policy documents that guide how evaluation is carried out within CTSI hubs

Moving the process forward: Key Questions What are the critical internal factors (within an AMC) that inhibit or facilitate DE (Clara Palfrey) 1. Culture, personnel, funding streams, institutional leadership, data systems, etc. What are the larger contextual issues that inhibit or facilitate DE (Julie Rainwater) 2. CTSA grant requirements, NCATS policies and priorities, common metrics, IOM report, etc. What steps can proponents of DE take within their institution to increase interest and create a hospitable environment for DE (Gaurav Dave) 3. Planning processes, support systems, cultivation, etc. How should we go about measuring an organization’s readiness for DE (Doug Easterling) 4. Pre-conditions for innovation and organizational learning, measurable constructs, possible tools