Capturing and understanding impact Dr Ian Viney Medical

Capturing and understanding impact Dr Ian Viney Medical Research Council

Why the interest in capturing details of research impact? • Analysis/learning • • Accountability • • Efficient and effective use of funds Government, the public etc. Advocacy • • To better understand research activity, what leads to impact – potential to maximise return Explaining delivery, making the case for continued funding Allocation • Prioritising research areas for support Developing a research evaluation framework http: //www. rand. org/content/dam/rand/pubs/research_briefs/RB 9700/RB 9716/RAND_RB 9716. pdf

Measurement Challenges • Causality – link between inputs and outputs/impact non-linear and complex (“chainlink”/”payback” models) • Attribution – how much of the benefit results from the specified research input, as opposed to other research and non-research inputs (e. g. marketing, “spillovers” etc. ) • Cross country effects – collaboration, cofunding, mobility of researchers • Timescales – research might take decades to lead to impact, premature measurement will over-emphasise policies encouraging research that brings short-term benefit The Benefits from Publicly Funded Research (Martin and Tang, 2006) https: //www. sussex. ac. uk/webteam/gateway/file. php? name=Fac-BRM-UMIP&site=25

Pathways to Impact Input Outputs/Outcomes Generation of new Knowledge/ publication Trained people Funding for Research and Training Impacts (Academic/Economic/Social) Improvements to health (living longer and with better quality of life) Academic impact (effects on further research including other disciplines) Development of collaborative networks Improving the performance of existing businesses Intellectual property/ Licensing Creating new businesses (that contribute to economic growth and further R&D) Research materials/ Technologies Influences on policy & practice Development of new products/processes Dissemination of research Delivering highly skilled people to the labour market Attracting R&D investment (from global business and non-UK funding sources) Improving public policy and public services (including the NHS) Engaging public support for medical research

Who funds health research, where, how much and on what? 2017 ? http: //worldreport. nih. gov/ 2014 HRCS analysis will show over £ 8 bn spent on health relevant research in the UK www. hrcsonline. net

Strategic Evaluation Research Studies • The MRC has encouraged academic researchers to take an interest in designing new approaches to evaluating impact – three rounds of funding since 2012 • This research should improve the current estimates of the economic return and more fully exploit the growing body of high quality data about output. • In 2014 this new “science of science policy” programme grew to seven small studies • We plan to hold a new workshop in 2015 to consider the next phase of this work MRC Call for proposals to understand better the link between research and impact http: //www. mrc. ac. uk/Fundingopportunities/Highlightnotices/MRCEconomic. Impact/MRC 009122

Medical Research: What’s it worth? (2008) A hybrid macro/micro economic study • Consistent time series for medical research funding in cardiovascular disease and mental health research from 1975 -1992 • Clear conceptual framework relating to GDP gain from “spillovers” • Estimation from literature of the magnitude of these effects • Development and application to CVD and MH, of a ‘bottomup’ approach to estimate health gain in terms of QALYs • Analysis of UK guidelines to provide indicators of elapsed time (lags) and proportion of benefits attributable to UK • Suggestions for developing research agenda Strong quantitative argument for investment in medical research http: //www. wellcome. ac. uk/About-us/Publications/Reports/Biomedicalscience/WTX 052113. htm

Net monetised health gain from medical research: Cardiovascular disease (CVD) New medical interventions • 22 million patients treated for CVD between 1985 -2005 • Total £ 2 billion support for CVD research (19751992), of which MRC provided £ 170 M Public/Charity Investment in Research directed to CVD • 12 guidelines formalise treatment for CVD • 17% of the knowledge cited in these come from UK research • Taking into account polytreatment a total of 2. 7 M QALYs generated, worth £ 69 billion (using value of £ 25 k per QALY) • Incremental costs of delivering these interventions (£ 16 billion) Patients treated Internal rate of return = 9% Net health gain £ 53 billion

Wider economic gains (spillovers) from public R&D: Improved UK based methodology Pharmaceutical/biomedical Sector • Increased R&D investment in the UK • Improved absorptive capacity leading to the creation of new knowledge and commercial gain Public/Charity Health R&D • New processes/products, patents • Expertise and entrepreneurship through • Increased collaboration with academia (modified Direct by proximity to discovery) networking and collaboration • Cutting edge infrastructure • New start up companies Ind i rec t Social return to economy • Improved productivity (better/new processes) • Reduction in costs for products and services e. g. NHS • Market entry decisions (new competitors, attracting new businesses) • Other spill-overs Internal rate of return = ? %

Elapsed time and translation Definition of specific points in the translational pathway to allow elapsed time to be measured Comparison of several case studies to identify what factors had the largest role in speeding up or slowing down translation “How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice. ” Hanney et al. http: //www. ncbi. nlm. nih. gov/pub med/25552353 http: //blogs. biomedcentral. com/on-health/2015/03/09/impact-impact/

Influence on policy and practice • For the UK citations in the guidelines issued by the National Institute for Clinical Effectiveness (NICE) are most influential – often used as a proxy for research being introduced into practice • We have analysed the extent of MRC’s contribution to guidelines using a highly quantitative approach, this can be done using digital versions of guidelines and Pub. Med records, our experience is that feedback from researchers is rather better at identifying changes to practice • Reports in Researchfish tend to be examples of substantive influences on recommendations within these guidelines, rather than simply linking research that is cited within the policy document – we have the advantage of the expert view of researchers – tapping into the links that matter http: //bmjopen. bmj. com/content/2/2/e 000897. full. pdf+html

MRC Research is saving lives now… • £ 125 m gain (50% NHS, 50% wider economy) in 3 years as cooling therapy limited brain damage in 3, 000 oxygen-deprived newborns • £ 32 m p. a. saved from shorter radiotherapy courses for breast cancer (£ 24 m to the NHS, £ 8 m to patients and families) Image © UK TOBY Cooling Register 2011 • Abdominal Aortic Aneurysm screening saves 6000 lives each year (at a cost of £ 2, 300 per life) • Immediate CT imaging to diagnose acute stroke generated 6, 000 more QALYs and NHS savings £ 156 m-£ 312 m each year • Ten REF 2014 impact case studies from Main Panel A were estimated to generate £ 2 bn impact in total health benefit

Analysis of REF impact case studies • Impact of HEI research (funded by Government, charities, industry and own funds is considerable, diverse and fascinating Underpinning research is multidisciplinary, and the social benefit arising is multiimpactful • • Different types of HEI specialise in different types of impact • UK HEIs have global impact • Evidence supporting impact cases was diverse and inconsistent, development of robust impact metrics is unlikely • Analysis would be better supported by more structured and consistent information http: //www. hefce. ac. uk/pubs/rereports/Year/2015/analysis. REFimpact/

Defined, structured and prospectively collected output information • Trace interactions between the producers and users of research, establishing evidenced links across as many steps as possible • Publications may have a minor impact in transferring knowledge to industry, people exchanges and problem solving (collaboration) are more important • Similarly focussing on hard commercialisation metrics (spin outs, patents etc. ) miss a large range of activities (e. g. policy influences) • Challenge is to capture robust details of collaborations and other processes effectively (using unambiguous categories, clear guidance etc. ) • Failures also have to be factored in (e. g. lack of productivity, absence of output), and so effort has to be made to capture feedback from all eligible researchers • Presents an opportunity to signal to the research community the outcomes that funding agencies value

Funders collaborate to collect research outputs Over 80 research funders, 20 research organisations Prospective tracking of outputs from research Researchers provide online feedback about output and attribute to funding Currently £ 4. 5 bn of new awards added each year, 1. 1 million outputs linked to £ 40 bn of research funding Validation against and enrichment with other data Wide variety of outcomes captured Data publicly accessible and provided for re-use by Universities e. g. via Gt. R Open Access Communicate benefits of research to public and policymakers – “Finding aid” for case studies Evaluation, strategy development, and crossfunder collaboration – Comparison/benchmarking What leads to impact? (science of science policy) – Modelling, analysis, tracking progress Quantitative and qualitative information Twitter @MRCe. Val http: //www. mrc. ac. uk/Achievementsimpact/Researchfish/index. htm

“Investable opportunities” • Over 600 projects developing new products or interventions - at all stages of development have been reported • These are also categorised in terms of product type (e. g. device, drug, behavioural intervention etc. ) • 110 have reached the market since 2006 • For the first time we can track all of these, noting which are active, closed, seeking support, and which are progressing • 75% of projects are prior to the “valley of death”, 25% after • Around 80 would benefit from further investment – so could be “investable opportunities” • The first year in which we published this data (2011) the three hundred development projects were used by the science minister as an example of the untapped potential that increased funding for the “biomedical catalyst” initiative could unlock • The biomedical catalyst is now a £ 180 m joint MRC/Technology Strategy Board initiative, having received an additional £ 90 m funding from Government “Valley of death”

MRC Researchfish 2012 Collaborations • Recipients of 61 per cent of awards reported that they had productive collaborations and partnerships between 2006 and 2012 • The majority of partners involved in these collaborations were with academia (58 per cent), followed by the public sector (16 per cent), and then the private sector (eight per cent) and hospitals (eight per cent) • Seven per cent of awards were highly collaborative, reporting active interactions to more than ten partner organisations • Private sector collaborations reported involved 500 unique companies worldwide • The top five private sector collaborators (GSK, Pfizer, Asta. Zeneca, Novartis and Merck) in total provided £ 72 m in direct financial support to MRC groups across 177 collaborations between 2006 and 2012. “In kind” contributions were varied and substantial

Research infrastructure used by UK researchers • • Take up across research councils in 1 st data collection variable – compliance is 73 -87% Reporting in this section currently low – 8. 5% (1500 researchers, 5000 attributions) Work using some facilities supported by multiple councils 940 unique facilities identified

MRC research attracting inward investment 65% of MRC awards leverage additional funding (from all non-MRC sources) within 4 years, and this leverage (in orange) now exceeds 120% of the MRC investment (blue). MRC developed a structured and prospective approach to capturing research output which is used to track progress, productivity and quality Important to understand how research leads to impact Challenging as research is longterm, there are many inputs and outputs are varied We have run this approach (now called “Researchfish”) since 2008 and have developed a detailed view of research output The approach is now used by 100 UK funding agencies which has built a dataset of more than 1 million outputs linked to £ 40 billion of research investment

Interdisciplinarity Areas of publication for 53, 000 papers arising from MRC research (2008 -2013) represented using “Science Overlay Maps” Rafols, Porter and Leydesdorff (2010)

Approaches to measure inter-disciplinarity Grant Applications • • It may be possible to categorise grant applications using text analysis (applying learning from the analysis of REF 2014 impact case studies, Uber. Research Ltd will apply cuttingedge text mining approaches to the analysis of RCUK grant applications Proposals to work in/across fields, success rates for subjects Publication Outputs Cited articles • • Categorisation of publications into subject areas is well established Is the knowledge generated relevant to diverse fields? Citing articles As above using the subject categorisation of articles cited in these publications Has the work drawn on results from different fields? • • As above using the subject categorisation of articles citing these publications Has the output been read/utilised by readers in different fields (diffusion of knowledge)?

Anatomy of a case study – structured comprehensive research output information Cytosponge costs around £ 25, and is done by nurses in a GP clinic, rather than endoscopy which costs at least £ 400, and is carried out in hospital. Industry collaborations exploring new hypothesis for development of BE and potential therapeutic approaches Charity and overseas funders are contributing to the clinical development of the device

How MRC has used the data • The data provides quantitative and qualitative evidence of the progress, productivity and quality of MRC research and we have contributed to over 16 MRC publications in the last four years • We rely on this evidence to compile the case to Government during exercises to allocate the science budget (“Spending Reviews”) • In 2010 the MRC data helped persuade Government that the budget for medical research should be protected in real terms against a backdrop of cuts, and that the strategy for life sciences should be placed at the heart of the UK’s plan for economic growth. The MRC budget was the only research council budget to be protected in this way • The data help us to measure progress against the MRC’s Strategic Plan, Research Changes Lives. The 2012 midterm progress report contained 220 achievements occurring between 2009 and 2012 harvested from the MRC dataset. • We also return the data to universities to scan for case studies for the REF • We are using the information actively in preparations for the 2015 spending review

Next steps • Encourage global open access to authoritative sources on public and charity funded research • Continue prospective collection of output data, and providing this as openly accessible • Integrate with ORCID and assess other identifiers • Link to sources such as IPO, clinical trial registries, Ethos etc. • Improve ease of use • Focus funder requirements for output information • Expand support for academic research on the science of science policy

Thank you evaluation@headoffice. mrc. ac. uk @MRCEval