The Collection and Validation of Financial Performance Data

The Collection and Validation of Financial Performance Data within a Voluntary Establishment Survey John Forth (NIESR) Robert Mc. Nabb (Cardiff University) ICES-III (June 2007, Montreal) National Institute of Economic and Social Research

Workplace Employment Relations Survey • National survey mapping employment relations in workplaces across Britain • Surveys in 1980, 1984, 1990, 1998 & 2004 • Core = cross-sectional survey of ~ 3, 000 workplaces, achieving response rates of ~ 70% in each year (voluntary) • Data collected primarily from “senior workplace manager responsible for HR issues” via 2 -hour face-to-face interview

Demand for financial data • Interest in the impact of employment practices on workplace performance – Training and workforce development – Innovative job design – Trade union organisation – Equality and family-friendly arrangements • Requires data on workplace practices, contextual characteristics and workplace performance (ideally repeated measures over time)

Existing data in WERS 1980 -98 • Manager’s subjective assessment of workplace productivity and financial performance Compared with other establishments in the same industry how would you assess your workplace's labour productivity: 1) A lot better than average 2) Better than average 3) About average for industry 4) Below average 5) A lot below average 6) No comparison possible 7) Relevant data not available

Existing data in WERS 1980 -98 • Advantages: – High item response rate (~90%) – Low cost • Concerns: – Limited knowledge of HR respondent – Few workplaces ‘below average’ – Limited response categories – Common rater bias – Lack of clarity about the underlying construct – Cannot quantify ‘impact’

Developments in WERS 2004 • Financial Performance Questionnaire • Four-page self-completion questionnaire issued after face-to-face interview for completion by finance manager • Questions based on ONS Annual Business Inquiry * Turnover * Employment costs * Purchases * Capital stocks * Capital expenditure * R&D activity • Target: completion at workplace level

WERS FPQ • Accepted by managers at 90% of the 1, 757 trading sector workplaces • 792 completed questionnaires (45% effective response rate) • Probability of response (Willimack and Nichols, 2001) : – Comprehension of request – Retrieval of information – Assessment of priorities – Release of data

Probability of response Comprehension: • HR mgr 47% • General mgr 42% • Finance mgr 66% Retrieval: • Records kept 46% • Not kept 26% Assessment of priorities: • Accepted SEQ 52% • Refused SEQ 26% Data release: • Single • Head Office • Branch site 52% 57% 39%

WERS FPQ • 626 questionnaires (79%) at workplace level – Single UK site – Head office – Branch site 97% 61% 72% • Item non-response low (<10%) on most items – – – Turnover Employment Labour costs Purchases Capital expenditure Value of assets 96% 94% 98% 91% 81% 58%

Validation of FPQ data • Compare with ONS Annual Business Inquiry • Collects data, inter alia, on gross output, intermediate inputs, employment and employment costs • Thus provides measures of productivity and profitability • Sample drawn from the official business register (IDBR), in common with WERS • Census of large enterprises (250+ emps. ); sample survey of smaller enterprises • Data usually provided at enterprise level

Correspondence of FPQ and ABI • Key question: how does the productivity data in the FPQ and ABI compare? • Provides an indication of data quality in FPQ (compared with official source) • Also indicates reliability of firm-level returns as proxy for site level performance • Workplaces available for this analysis: – 357 with output/head from ABI and FPQ – 82 where ABI and FPQ both at site-level – 206 where only one source at site-level

Output/head in FPQ and ABI All sites excluding >95 th percentile (n=336) Correlation = 0. 74

Output/head in FPQ and ABI Site level in both surveys, excluding >95 th percentile (n=76) Correlation = 0. 89

Output/head in FPQ and ABI Site level in one survey only, excluding >95 th percentile (n=194) Correlation = 0. 70

Other measures • Value-added per employee: – Both at site level: r=0. 81 – Only one at site level: r=0. 41 • Profit per employee: – Both at site level: r=0. 78 – Only one at site level: 0. 35

Conclusions at this point • Data from the FPQ corresponds well to ABI data when both returns are at site level • The data do not compare so well when there is a mismatch in the unit of observation • The degree of correspondence is ~ inversely related to the breadth of the aggregate return

Site-level data • Take site-level returns from FPQ or ABI • Availability of output/head: – FPQ or ABI – FPQ only – ABI only – TOTAL 114 419 120 653 (37% of 1, 757 trading sector)

Comparison with subjective data • Examine convergent validity by comparing sitelevel LP with the subjective measure • Use site-level LP relative to average in four-digit SIC Class. Below average/ lot below About average Above average Lot above average 25 th ptile 0. 55 0. 67 0. 64 0. 59 Median 0. 83 0. 95 1. 04 1. 16 75 th ptile 1. 20 1. 64 1. 65 2. 69

Comparison with subjective data • Discriminant validity: – subjective measures of workplace productivity and profitability do not discriminate well between objective measures of these items • Construct validity: – identically-signed associations with a limited range of independent variables (e. g. unionisation) but fewer significant associations when using the subjective measure

Overall conclusions • New source of objective data on workplace performance in WERS 2004 • Response rate reasonable for voluntary survey and given available resources • Positive attributes: – Low item non-response – Mostly site-level observations – Favourable comparison with ABI data • Negative attributes: – Survey non-response biases – Firm-level responses of questionable value – Only one time point at present

Overall conclusions • Evaluation of subjective measure: – – Low non-response Convergent validity: weak Discriminant validity: very weak Construct validity: high • Triangulation important

Further information • Detailed academic papers available on request from: j. forth@niesr. ac. uk • Further information on WERS 2004: http: //www. wers 2004. info

Acknowledgements & disclaimers • • • We gratefully acknowledge the advice and support provided in the early stages of WERS 2004 by the other members of the Specialist Advisory Team on Performance and Technology (Prof Stephen Machin and Prof Richard Harris) We also acknowledge the WERS Research Team and the WERS sponsors (DTI, ESRC, Acas and PSI) as the originators of the WERS 2004 data, and the ONS as the originators of the Annual Respondents Database and the Annual Business Inquiry data. We would also particularly like to thank Felix Ritchie and Rhys Davies (Business Data Linking, ONS) for their work in matching WERS to the ARD The ABI data provided by ONS is Crown copyright and reproduced with the permission of the controller of HMSO and Queen's Printer for Scotland. The use of the ONS statistical data in this work does not imply the endorsement of the ONS in relation to the interpretation or analysis of the statistical data. This work uses research datasets which may not exactly reproduce National Statistics aggregates None of these organisations bears any responsibility for our analysis or interpretation of the data.