Evaluating RD Programs RD Evaluation Planning Concept Mapping

Evaluating R&D Programs: R&D Evaluation Planning Concept Mapping and Logic Modeling • • • Presentation to the Washington Research Evaluation Network June 2008 • • • William Trochim Cornell University wmt 1@cornell. edu

Questions • What new useful and applicable approaches and tools for ex-ante evaluations have been uncovered? • Which agencies have used these tools? • How can these tools be matched with different levels of decision making? • Can systems-level analysis address gaps and assist in science and technology decision making at the federal level?

Overview of Evaluation Planning Process • • • Concept Map Logic Model Evaluation Questions Measurement Development Evaluation Design Implement Evaluation

Where Has Approach Been Used • NIH/NCI Transdisciplinary Tobacco Use Research Centers (TTURCs) • CDC, Prevention Research Centers (PRCs) • NIH/NIAID/DMID, Research Centers of Excellence (RCEs) • NIH/NIAID/DAIDS, AIDS Clinical Research Networks • NIH/NCRR/CTSA, Weill Cornell Clinical and Translational Science Center

DMID Research Centers of Excellence (RCEs) PLANNING: Identified the focus prompt and participants “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should. . . ” IDEA GENERATION: Eliciting knowledge and opinion Brainstorming : • 279 stakeholders were invited to participate. • Participants generated 248 statements with 117 unique visitors hitting the site. • 100 statements were derived from the Literature Review. • The planning group edited statement list for relevance and representativeness. • Planning Group approved final set of 94 ideas. STRUCTURING: Data collection to build the conceptual framework Sorting and Rating of Ideas • 279 participants were invited to rate the 94 ideas on relative Importance and Expected Progress from 2003 -present • 74 people (27%) contributed their input on the ratings. • 33 people invited to organize the 94 ideas into conceptually similar categories. • 24 people (72%) participated in the conceptual sorting of the ideas. ANALYSIS: Series of multivariate analyses

The point map shows all the elements in relation to one another. value and reward original ideas and innovative technologies (4) “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should. . . ”

Conceptually similar ideas are in close proximity value and reward original ideas and innovative technologies (4) demonstrate its flexibility in addressing emerging infectious disease research needs. (19) have the capacity to rapidly move into areas of growing importance. (56)

Conceptually different ideas are further apart define who the stakeholders are once product development moves into the pipeline (72) value and reward original ideas and innovative technologies (4) create training that is relevant to and integrated with the goals of the strategic plans. (21) “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should. . . ”

The cluster map organizes ideas into groups Research Mission 59 11 20 60 618 1224 39 47 54 3 Innovation and Flexibility 19 52 13 94 Translational Activities 30 72 46 35 57 40 63 80 70 71 45 Emergency Response 10 218 7 Training and Recruitment 16 55 26 76 56 4 34 79 62 68 51 23 67 43 Capacity and Infrastructure 58 6538 75 22 29 48 Collaboration and Communication 28 31 64 36 5 69 1 32 73 Integration and Synergy 37 90 66 92 15 83 82 84 89 33 Funds Management 53 42 74 93 41 85 819 2 86 77 61 25 49 14 50 88 17 27 87 78 44 91 Management

Each cluster retains the details: For example, Training and Recruitment promote and encourage opportunities for mentoring junior faculty/researchers in the field of biodefense and emerging infectious disease research. (7) encourage the participation of post-docs, students, and other non-PI scientists at each RCE's regional meeting. (8) recruit qualified students, trainees, women and minorities. (10) help new investigators obtain funding. (16) create training that is relevant to and integrated with the goals of the strategic plans. (21) develop more standardized training programs across RCEs by looking at successful programs at each RCE and developing some best practices. (26) develop an expanded cadre of new researchers, clinicians, and technical personnel who can help lead the national biodefense mission. (45) supplement RCE activities by applying for direct training supports through T, K and F awards. (55) establish training programs with pharmaceutical company partners to allow training of scientists in drug discovery and non-clinical development. (68) develop policies, criteria and processes for selecting career development candidates, including special efforts to recruit qualified women and minorities. (76) bring new people into the area of biodefense and emerging infectious disease research. (80)

DMID RCE Final Conceptual Framework Research Mission Innovation and Flexibility Translational Activities Capacity and Infrastructure Collaboration and Communication Emergency Response Integration and Synergy Training and Recruitment Management Funds Management …contains all the details and provides a conceptual framework.

Concept Map with Regions Research Mission Outcomes Translational Activities Innovation and Flexibility Capacity and Infrastructure Activities Collaboration and Communication Emergency Response Training and Recruitment Integration and Synergy Management Funds Management Outputs

Logic Model for RCE Program Inputs Activities Outputs Outcomes Manage Funds flexibly and efficiently Innovative & Flexible response to changing needs and opportunities RCE Program Funding Establish Management structures & processes Translate and Apply Science to Practice Leveraged funds from elsewhere (Integration & Synergy) Capacity to serve as a regional resource Leadership with expertise (Integration & Synergy) Collaboration & Communication Expanded cadre of investigators (Recruit and Train staff) Build Infrastructure Biodefense Priorities Synergy between Centers (Integration & Synergy) Recruit & Train staff Conduct Research on Category A, B, C organisms Research results on Category A, B, C organisms Impact Scientific Knowledge on Category A, B, C organisms (Research ) Readiness to Respond in an Emergency Product Development (Translational Activities) Better response to infectious diseases that threaten humans

TTURC Concept Map Health Outcomes Policy Implications Communication Improved Interventions Outcome Health Impacts Translation to Practice Communications Science & Models TD Research Institutionalization Structure Professional Validation Recognition Methods Training Collaboration Transdisciplinary Integration Scientific Integration Process Collaboration Publications

TTURC Logic Model External Recognition And Support Communication Internal Recognition And Support Professional Validation Communication Collaboration Training Publications Methods Collaboration Health Impacts Immediate Markers Health Outcomes Interventions Science & Models Transdisciplinary Integration Policy Implications Translation To Practice Scientific Integration Intermediate Markers Long-Term Outcomes

CDC PRC Concept Map Engage the Community Diversity & Sensitivity Research Methods Relationships & Recognition Training Research Agenda Technical Assistance Core Expertise & Resources Active Dissemination

From Map to Logic Model Inputs Activities Outputs Outcomes Core Expertise & Resources Engage the Community Relationships & Recognition Diversity & Sensitivity Training Research Methods Research Agenda Technical Assistance Core Expertise & Resources Active Dissemination

Draft logic model Inputs Activities Community Advisory Board Engage the Community National, Regional or Local Health Priorities and Health Disparities PRC Capacity • Core expertise & resources • Diverse faculty/ staff • Faculty/staff sensitive to community issues • Facility • Communication & data systems Conduct Core & Other Research Using Sound Research Methods Test Innovative Strategies Disseminate Research Findings Outcomes Translation of Research To Practice Widespread Knowledge of Effective Interventions Conduct Training Trust Provide Technical Assistance Widespread Use of Effective Interventions Research and Other Publications Recognition Establish Research Agenda Relationships With Community Partners, Other PRCs, and CDC Outputs Skilled Workforce Improved Community Health and Reductions in Health Disparities

Logic Model INPUTS PROGRAM ACTIVITIES OUTCOMES OUTPUTS Evaluation National, Local, or Regional Health Priorities And Health Disparities PRC Community Committee PRC Capacity • Human resources: core expertise, diversity, sensitivity • Facility • Communication & data systems • Administrative capacity • Evaluation expertise Relationships with State & Local Health Departments, Community Partners, University Partners, Other PRCs, and CDC Engage the Community Establish Research Agenda Conduct Core and Other Research Using Sound Research Methods Train/Mentor/P rovide Technical Assistance to: • Researchers • Practitioners • Students • Community Members Programs/ Interventions Translation of Research to Practice and Policy Research/ Evaluation Findings Communicated & Disseminated in: • Publications • Presentations • Media • Reports Widespread Use of Effective Programs and Policies Improved Community and Population. H ealth and Elimination of Health Disparities Enhanced Community Capacity for Prevention Trainees and Technical Assistance Recipients Skilled Public Health Professionals Motivating Conditions for Developing and Maintaining Relationships (e. g. , Trust) Expanded Resources Recognition

AIDS Clinical Research Networks Map Relevance to Participants Biomedical objectives Community Involvement Scientific Agenda-setting Resource Utilization Communication, Collaboration, Harmonization DAIDS Policies and Procedures Operations and Management

Inputs Activities Community Involvement Community input (Community Involvement) Appropriate human and infrastructure Resources are in place DAIDS Policies and Procedures Scientific Agenda Setting processes Collaboration and Communication (Communication, Collaboration, Harmonization) Operations and Management processes are in place Outputs Recruitment and retention goals met (Community Short Term Outcomes Longer Term Outcomes/ Impact Involvement) Increased capacity of developing world sites (Resource Utilization) Integrated use of developing world sites (Resource High quality scientific results and increased knowledge (Biomedical Objectives) Treatment and prevention measures (Biomedical Objectives) Utilization) Scientific research plan and priorities (Scientific Agenda Setting) Harmonized systems and procedures (Communication, Collaboration, Harmonization) Scientific agenda is Relevant to Participants Results Published (Biomedical Objectives) Biomedical Objectives: HIV/AIDS mortality and morbidity reduced

Conclusions • • • Participatory and Collaborative Integrative Mixed Methods Systems-Based Approach Comprehensive Conceptual Model Foundation for e. Evaluation

Relevant Literature • Trochim. W. , Marcus, S. E. , Mâsse, L. C. , Moser, R. P. , Weld, P. (2008). The Evaluation of Large Research Initiatives: A Participatory Integrative Mixed-Methods Approach, American Journal of Evaluation, 29, 1, 8 -28. • Anderson, L. A. , Gwaltney, M. K. , Sundra, D. L. , Brownson, R. C. , Kane, M. , Cross, A. W. , et al. (2006). Using concept mapping to develop a logic model for the prevention research centers program. Preventing Chronic Disease: Public Health Research, Practice and Policy, 3(1), 1 -9. • Trochim, W. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1), 1 -16. • Trochim, W. and Kane, M (2005). Concept mapping: An introduction to structured conceptualization in health care. International Journal for Quality in Health Care, 17, 3, June 2005, 187 -191. http: //www. socialresearchmethods. net/research/cm. htm

Further Information • William Trochim – wmt 1@cornell. edu – http: //www. socialresearchmethods. net/ • Concept Systems Incorporated – http: //www. conceptsystems. com/