Pilot and feasibility studies Garry Tew garry tewyork

Pilot and feasibility studies Garry Tew garry. tew@york. ac. uk BOA Orthopaedic Surgery Research Centre

Pilot or Feasibility? • Term ‘pilot study’ - often synonymous with ‘feasibility study’ • One methodological review could find “no formal methodological guidance as to what constitutes a pilot study” (Lancaster et al. , 2004) • Definitions are not distinct and can vary between funding bodies • NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) definitions are very helpful: http: //www. netscc. ac. uk/glossary/

What is a feasibility study? • A feasibility study is a piece of research done before a main study to estimate important parameters that are needed to design the main study. • “Can this study be done? ”

What is a feasibility study? For example: • Willingness of participants/ clinicians • Number of eligible patients • Follow-up rates, adherence/ compliance • Standard deviation of outcome measure • Possibly defining/ designing a suitable outcome measure • They do not necessarily evaluate the outcome of interest • Do not need to have a primary outcome and the usual sort of power calculation is not normally undertaken

What is a pilot study/trial? • A pilot study also precedes the main study • A version of the main study which is run in miniature to test whether study components can all work together • Although MRC state: “A pilot study need not be a ‘scale model’ of the planned main stage evaluation, but should address the main uncertainties that have been identified in the development work” • Ensure recruitment, randomisation, treatment and follow-up assessments all run smoothly • Includes the assessment of the primary outcome

What is a pilot study/trial? • “… too small to have a reasonable chance of detecting a minimum important effect size” or • “… a study that is evaluating an incompletely developed intervention. ” Torgerson and Torgerson (2008)

Pilot trials • Two types of pilot trial: • Internal – part of the main study (a run-in to the main study; data from pilot phase contribute to the final analysis) • External – independent of the main study (data from the pilot are analysed and set aside)

Why do an internal pilot trial? • To test recruitment, trial processes, etc… • Sample size recalculation: The main study is planned and a sample size calculation is performed on the best available data. The internal pilot is conducted on a prespecified number of patients entering the study and then the sample size is recalculated based on the estimates obtained (e. g. AIM trial). • Allows change towards a more appropriate sample size within the same overall study • However, many other key aspects of the study design cannot be changed following an internal pilot (e. g. primary outcome, intervention design) (Lancaster et al. , 2004)

What is the purpose of an external pilot trial? • Test the feasibility of a main study both logistically and financially, reducing wastage of resources and time • Gives advance warning of where a main study could fail • Doesn’t guarantee the success of a main study but increases the likelihood

Reasons for doing an external pilot trial Design issues: • Help inform the protocol • Assess whether the protocol is realistic and workable • Regulatory approval problems • Local politics • Randomisation procedure • Recruitment and consent procedures • Inclusion and exclusion criteria

External pilot: reasons continued… • Test the research instruments / outcome measures • e. g. help select the most appropriate primary outcome measure • Test the data collection forms • Help determine the sample size and power calculation – although some authors do recommend this with caution (Kraemer et al 2006)

External pilot: reasons continued… People issues: • Assessing willingness to take part or likely response rates • Test the acceptability of the intervention • Assess whether those collecting the data can do so accurately • Assess how long it take for them to do so and acceptability of this • Assess whether they all collect the same things in the same way!! • i. e. – training and /or documentation issues • Training the researchers!

External pilot: reasons continued… Finance / Funding issues: • Determine what resources are needed for the main study including staff and materials • Convincing the funding bodies that your proposal for the main study is worth funding • Convincing the funding bodies that your research team are competent!

External pilots: problems with them • May be in different setting to main study and different issues may arise, over-estimating ability to do the main trial • Inaccurate predictions or assumptions may be made on the basis of the pilot study data • Should the data be included in and contribute to the main analysis? • In a quantitative study – usually not • In a qualitative study – usually are

External or internal? • External pilots will delay the start of any main trial – therefore an internal one may be more attractive • External pilots are sometimes difficult to get funded – however; • With internal ones it is not always possible to fully apply any lessons to the main trial

Study design for pilots? • Generally should be as similar as possible to the anticipated design for the main study • Should a pilot be randomised? • Non-randomised pilots may give misleading data on recruitment, retention, effect size etc…

Examples of piloting of surgical trials PROFHER • Pragmatic RCT of surgical vs. non-surgical treatment for proximal fracture of the humerus • Target n = 250 from 18 sites over 18 months (assuming 1 patient/month at each site) • Agreed with HTA to have recruited 88 patients in first 10 months • But… only 33 patients were recruited • Therefore the number of other sites to be set up was doubled and recruitment extended by 12 months

Examples of piloting of surgical trials SWIFFT • Pragmatic RCT of surgical vs. non-surgical treatment for scaphoid waist fractures • Target n = 438 from 17 sites over 30 months (assuming 1 patient/month at each site) • Agreed with HTA to have recruited 24 patients from 4 sites in first 6 months (+ 66% retained of these patients retained at 5 years) • But. . . only 18 patients were recruited • Therefore the team doubled the number of other sites to be set up (and trial is now recruiting to target)

Examples of piloting of surgical trials UK-FROST • Pragmatic RCT of physiotherapy vs. manipulation under anaesthesia vs. arthroscopic capsular release for frozen shoulder • Target n = 500 from 25 sites over 30 months • Agreed with HTA to have: - Recruited 24 patients from ≥ 4 sites in first 6 months - 12/25 sites set up to recruit within 6 months • Extended pilot (18 months): Assess extent of refusal of patients to participate (likely due to randomisation to physiotherapy) and refusal to continue with allocated treatment (again likely to be physiotherapy)

Other examples Acupuncture for pain and osteoarthritis of the knee (external pilot) • Pilot RCT of 30 patients randomised to acupuncture + usual care vs. usual care alone • Primary outcomes were to establish: recruitment rate, appropriate outcome measures, attendance levels for acupuncture, loss to follow-up, sample size for full RCT • Identified by database search n = 335, did not respond n = 235, passed first screening n = 78/100 (i. e. 23% of those initially identified) • WOMAC more sensitive to change than OKS • 136/150 (90%) appointments attended • Loss to follow-up: 0 vs. 1 at 3 months, 2 vs. 7 at 12 months • N = 350 needed for full trial Lansdown et al. (2009)

Where a pilot should have been done • AESOPS - evaluating the effectiveness and costeffectiveness of stepped care interventions for older hazardous alcohol users in primary care (aged 55 years and over). • ‘Positive’ patients identified by score of 8 or more using the Alcohol Use Disorders Identification Test (AUDIT) • Interventions delivered by practice nurses/ research nurses in GP practices • Estimated that 15 GP practices would recruit 500 patients. • Eventually rose to 30 GP practices – all doing ‘opportunistic screening’

Recruitment after 18 months

What happened? • Team panicked a bit… • Decided to pilot a few new practices using mail outs and more research nurses (PNs were struggling to commit adequate time alongside usual duties) • Seven of the original practices supplemented their opportunistic screening with mail out • 24 new practices set up to use mail shots only • Approx. 13285 screening packs put out in practices • Approx. 65000 screening packs mailed out • 21524 returned (27. 5%)

Effect of recruitment method change

Publications • Pilot trials are often not given sufficient publicity – often by the researchers as well as journals. • Researchers often state they have learned for the pilot trial but do not go into detail or concentrate on one element (van Teijingen and Hundley, 2001) • Journals are often criticised for not publishing pilots • Papers reporting pilot results can often be stigmatised or sidelined, which one author considers unfair and punishes the researcher for being methodical (Shuttleworth, 2008)

“Well designed and well conducted pilot studies can inform us about the best research process and occasionally about the likely outcomes” (van Teijingen and Hundley, 2001)

References • • • Kraemer HC, Mintz J, Noda A, Tinkelnberg J, Yesavage JA (2006) Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry 63 (May): 484 -489 Lancaster GA, Dodd S and Williamson PR (2004) Design and analysis of pilot studies: recommendations for good practice. Journal of Evaluation in Clinical Practice 10 (2): 307 -312 Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Goldsmith CH. A tutorial on pilot studies: The what, why and How. BMC Medical Research Methodology 2010, 10: 1 Arain M, Campbell MJ, Cooper CL and Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology 2010, 10: 67 Reynolds JA, Bland JM, Mac. Pherson H (2008) Acupuncture for irritable bowel syndrome –an exploratory randomised controlled trial. Acupuncture in Medicine 26 (1) 8 -16 Shuttleworth M (2008) Pilot Study- Testing the Study Before Launching [Online] Available at http: //www. experiment-resources. com/pilot-study. html [Accessed 27 Aug 2009] Lansdown, H. , Howard, K. , Brealey, S. and Mac. Pherson, H. (2009). Acupuncture for pain and osteoarthritis of the knee: a pilot study for an open parallel-arm randomised controlled trial. BMC Musculoskeletal Disorders, 10: 130 Tew, G. A. et al. (2015). The development and pilot randomised controlled trial of a group education programme for promoting walking in people with intermittent claudication. Vascular Medicine. Torgerson DJ and Torgerson CJ (2008) Designing Randomised Trials in Health, Education and the Social Sciences. 1 st ed. Hampshire Palgrave Macmillan van Teijlingen E. , Hundley, V. (2001) The importance of pilot studies. Social Research Update 35 van Teijlingen ER, Rennie A-M, Hundley V, Graham W (2001) The importance of conducting and reporting pilot studies: the example of the Scottish Births Survey. Journal of Advanced Nursing 34 (3): 289 -295