Auditing data management system Bruno Deprez Data Audit
Auditing data management system Bruno Deprez Data Audit officer SPC, Oceanic Fisheries Program
What is data management audit? • Definition: “…a formal, often periodic examination and checking of data or procedures to verify their correctness…” • “Audit” term is often used in financial and accounting sectors – To evaluate organization and people – To validate the official financial results of an organization before publication • Fishing data management audit purpose is different. The final objective is to improve the quality level of data entered in country in order to make the studies based on this data easier and better • To achieve this objective, we need to ensure the correctness and the coverage of the data, throughout systemic and non-systemic audit • This audit is only applied to logsheet data
Why audit data ? Good Data = Good Decisions • To make sure a decision or a strategy is right, the first thing to do is to make sure the data are representative – Is it correct ? – Is it complete ? • Usage of logsheet data at member country level – Level of exploitation by national fleet – Level of exploitation in your EEZ – Decisions on National Fisheries Management – National reports based on fisheries data • Usage of logsheet data at regional level – Level of exploitation in the region – Trends in Catch, Effort and CPUE – Stock assessments • Need to audit the data to make sure decisions and strategies are not based on biased information
Ramifications of unrepresentative data … Examples… • Under-estimating your domestic fleet catch… – Will not benefit in allocation issues … – Potential problems with input data to Regional and National assessments • Under-estimating foreign fleet catch in your EEZ… – Will not benefit from fees related to catch in zone • Not identifying misreported catches on the logsheets… – Will not benefit from fees related to catch in zone – Potential problems with input data to Regional and national assessments
Objectives of Data Audit • To ensure that data are standardized – Almost guaranteed by the use of TUFMAN system to enter data – Basic checks on the database to make sure it is used properly • To improve the quality and coverage of data available at national level and to the WCPFC – Data quality audit is applied to position/time, catch and effort – Coverage audit tries to identify missing logsheets – Both use extensively the VMS data and samples of paper logsheets • To improve data collection in country – Existence of data registration system – Timeliness and frequency of provision of logsheet data from companies – Level of liaison/cooperation with fishing companies to retrieve logsheets – Existence of a filing system to easily retrieve hard-copy or scanned logsheets
In-Country Data Audit Process Completeness Presentation of audit process Accuracy 1. Run the VMS reconciliation report 1. Run Data Quality Tests 2. Communicate missing logsheet list to the fishing companies 2. Modification of obvious errors into TUFMAN 3. Reception and entry of logsheets into TUFMAN 3. Use audit geographic system for warnings TUFMAN 4. Evaluation of completeness in audit report 1. Manual check of a sample of logsheets 3. Check logsheet flow from reception to archiving 4. Modification of warnings into TUFMAN 5. Evaluation of accuracy in audit report Presentation of audit report and conclusion
In-Country Visit • One week visit – Usually data audit takes 4 days – 5 th days is spent on discussion, and other SPC task in-country (upgrading tools like SLOPS, upgrading Tufman system, troubleshooting problems) • Fully funded by the Sci. COFish project • List of country to be visited in 2013 – Tuvalu / Kiribati / Solomon Islands / FSM / Fiji / Palau
The Data Quality Audit Tool • Stand-alone application to be linked to your Tufman database • Comes with a set of tests separated in three categories : – Position/Time • The tests will verify the accuracy of logsheets times and positions • E. g. Average speed between sets, Missing sets, Overlapping trips, etc – Catch • The tests will verify the accuracy of the catch part of logsheets • E. g. By Catch – Effort • The tests will verify the accuracy of the effort part of logsheets • Discrepancies in number of hooks • Generates pre-formatted Excel reports with test status and list of warning and error database entries • Comes with an embedded GIS option in order to visualize any trip on the map • The list of tests is not static and new ones can be easily added throughout the audit process • Some basic automated correction could be also applied on demand
Example : The average speed between sets
Example of a test: Use of limits The speed limits for a longliner in the tool are set as follows : – Raise a warning if the speed is above 11 kts – Raise an error if the speed is above 15 kts
Example : The average speed between sets
Example of a test: The detail part • 324 knots is obviously wrong • The error is on the longitude entered for Feb 7 th – 01733 E is wrong, it should be something around 173 E (this points ends up in Namibia, Africa) – Need to get back to the paper logsheet and see what was the right value
Use of GIS feature on error • This trip error is obvious • It is not always the case, and viewing the trip itself can help finding some answers • Let’s take a look at this example
Use of GIS feature on error
Use of GIS feature on error
Use of GIS feature on warning • Previous example was from the error report. Let’s take one from the warning report (much less obvious) • The SOLANDER 6 has been steaming at 11 -12 knots between 3 positions, it is above the warning threshold • Here the use of the GIS feature can help to ensure there is a position error on the logsheet
Use of GIS feature on warning
Use of GIS feature with VMS • In order to help the decision to modify or not a point, the VMS reconciliation can be used (when applicable) • This process is explained in details next on this presentation • Its main purpose is to compare the logsheets with the trips generated from VMS and find reconciliations
Use of GIS feature with VMS
Data Collection Audit • During in-country visit, the following aspect of data management will be checked – Data Registration • E. g any potential hard-copy logsheet going missing prior to data entry or submission to regional agency – Hard-copy data management • E. g does the responsibility matrix for hard-copy data management cover all work and match actual responsibilities – Database System • E. g Are there any database post data-entry checking • TUFMAN logsheets will be checked against randomly selected hard-copy logsheets
In-Country Data Audit Output • The output of an audit consists in – The audit conclusion containing the audit results • For each part of the audit (quality, coverage and data collection), the detailed status : FAILED, WARNING or SUCCESS • The global audit status • The summing up of the actions to be undertaken – The details of quality audit • For each quality test (average speed between sets, inland positions, etc) • The warnings and errors to be modified in Tufman – The detail of data collection • Comments about each logsheet manually checked • Comments about how to improve data collection – The detail of logsheet coverages per year, gear and flag • To be generated in Tufman using the new report
Systemic Data Audit • Tufman Reconciliation report is analysed during in-country visit • The same tool is used internally at SPC to compare the same VMS data with the logsheets sent by the member countries to SPC • Every quarter, an export will be sent to countries to highlight gaps in the logsheets received at SPC • Two main reasons for a gap – Data received in-country from fishing company but not sent to SPC – Data not received in-country from fishing company
Future Online Audit Tool • Following reconciliation example, data audit tool will be pushed to the cloud – It will be part of e. RECAP – It will use your Tufman backup sent to SPC – It will provide you quality status data on your Tufman system • In-Country visit will still be indispensable – To focus on key issues highlighted by the online audit – To audit logsheet management, which cannot be done online
Questions? Remarks?
- Slides: 24