Benchmarking Methodology Ral GarcaCastro Asuncin GmezPrez rgarcia asunfi

Benchmarking Methodology Raúl García-Castro Asunción Gómez-Pérez <rgarcia, asun@fi. upm. es> September 27 th, 2004 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 1 © R. García-Castro, A. Gómez-Pérez 1

What has been done? in D 2. 1. 1 Survey of Scalability Techniques for Reasoning with Ontologies Ontology Technology/Methods Measurement Desired attributes Weaknesses Comparative analysis. . . Benchmarking Evaluation Experimentation Continuous improvement Best practices Recommendations • Overview of benchmarking, experimentation, and measurement • State of the Art of Ontology-based Technology Evaluation 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 2 © R. García-Castro, A. Gómez-Pérez 2

What are we doing? T 2. 1. 4 Benchmarking methodology, criteria, and test suites GOAL: Provide a framework for benchmarking activities in WP 2. 1 Benchmarking results: Ontology tools: • Ontology building tools • Annotation tools • Querying and reasoning services • Semantic Web Services technology Benchmarkin g Methodology • Comparative analysis • Compliance with norms • Weaknesses • Recommendations on tools • Recommendations on practices General evaluation criteria: Benchmark suites for: Benchmarking supporting tools: • Interoperability • Scalability • Robustness • Workload generators • Test generators • Monitoring tools • Statistical packages • . . . 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 3 © R. García-Castro, A. Gómez-Pérez 3

What will be done? T 2. 1. 6: Benchmarking of ontology building tools Interoperability • Do the tools import/export from/to RDF(S)/OWL? • Are the imported/exported ontologies the same? • Is there any knowledge loss during import/export? • . . . Tools/Partners: Experimentresults: Experiment UPM . . . Benchmarkin g ontology building tools Benchmark suites: • RDF(S) Import capability • OWL Import capability • RDF(S) Export capability • OWL Export capability • . . . 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 test 111 NO test • • • test 222 OK test • • • test 333 OK test • • test. . . • • • . . . Benchmarking results: • Comparative analysis • Compliance with norms • Weaknesses • Recommendations on tools • Recommendations on practices Benchmarking supporting tools: • Workload generators • Test generators • Monitoring tools • Statistical packages • . . . 4 © R. García-Castro, A. Gómez-Pérez 4

Benchmarking methodology Methodology: Benchmarking process is: • Planned • Collaborative Inputs Task 1 Processes Outputs . . . Task n Methodology processes Plan Experiment Improve 1 Goals identification 7 Experiment definition 10 Report writing 2 Subject identification 8 Experiment execution 11 Findings communication 3 Management involvement 9 Experiment results analysis 12 Findings implementation 4 Participant identification 13 Recalibration 5 Planning and resource allocation 6 Partner selection 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 More Semantic Web oriented More KW oriented 5 © R. García-Castro, A. Gómez-Pérez 5

Benchmarking methodology Plan 1. - Benchmarking goals identification • Goals depend on the organisation’s vision, objectives, and strategies. 2. - Benchmarking subject identification • Analyse the current tools in the organisation. • Select, understand, and document the tool whose improvement would significantly benefit the organisation, according to: end user needs or expectations, organisational goals, etc. 3. - Management involvement • Inform the organisation's management about the benefits of the benchmarking study and its costs. • Management support is needed to proceed and when implementing changes based on the benchmarking. 4. - Participant identification • Identify and contact the members of the organisation that are involved with the selected tool. • Select and train the members of the benchmarking team. 5. - Benchmarking planning and resource allocation • The planning must consider time and resources. • The planning must be integrated into the organisation's planning. 6. - Benchmarking partner selection • Identify, collect, and analyze information about the tools that are considered the best. • Select the tools to benchmark with and make contact with someone in their organisations. • The partner organisations may not belong to KW. Not all ‘best in class’ tools are developed by KW partners. 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 6 © R. García-Castro, A. Gómez-Pérez 6

Benchmarking methodology Experiment 7. - Experiment definition • Determine the experimentation plan and method. • Define the experiment that will be performed. The experiment must collect not just the data on the performance of the tools but the reasons of this performance. • Communicate the partners the experimentation plan and method and agree on it. 8. - Experiment execution • Perform the experiment according to the experimentation plan and method. • The collected data must be documented and prepared for analysis. 9. - Experiment results analysis • Compare the results obtained from the experiments and the practices that lead to these results. • Document findings in a report, including the best practices found (if any). 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 7 © R. García-Castro, A. Gómez-Pérez 7

Benchmarking methodology Improve 10. - Benchmarking report writing The benchmarking report must provide an understandable summary of the benchmarking study with: • An explanation of the benchmarking process followed. • The results and conclusions of the experiments. • The recommendations on improving the tools. 11. - Benchmarking findings communication • Findings must be communicated to all the organisation (including identified participants) and to the benchmarking partners. • Collect and analyze any feedback received. 12. - Benchmarking findings implementation • Define a planning for the implementation of the benchmarking findings. • Implement the necessary changes in order to achieve the desired results. • Periodically monitor the benchmarked tool. 13. - Recalibration • Recalibrate the benchmarking process using the lessons learnt. • The benchmarking process should be repeated forever in order to obtain a continuous improvement. 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 8 © R. García-Castro, A. Gómez-Pérez 8

Benchmarking Methodology Raúl García-Castro Asunción Gómez-Pérez <rgarcia, asun@fi. upm. es> September 27 th, 2004 18/12/2021 Benchmarking Methodology. Sep 27 th, 2004 9 © R. García-Castro, A. Gómez-Pérez 9