On the Usefulness of Different Expert Question Types

On the Usefulness of Different Expert Question Types for Fault Localization in Ontologies DX’ 19 Patrick Rodler, Michael Eichholzer University of Klagenfurt

Motivation Faults occur frequently during the evolution of ontologies Reasons / Fault-promotive factors: e. g. • collaborative development • vast ontology size • high complexity • automated tools • expressive logics Faults of semantic nature (e. g. , inconsistency, wrong entailments) particularly critical • e. g. , medical semantic system suggests wrong therapy for patient Goal: Localize (semantic) faults in ontologies efficiently! DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 2

Which Problem Domain Do we Address? Fault Localization in Ontologies: Given: ontology that violates given requirements Task: find diagnosis requirements = consistency, no unsat classes, no wrong entailments, … diagnosis = irreducible set of axioms explaining requirements violation used for ontology repair Problem: • often substantial number of competing diagnoses, • all leading to semantically different repaired ontologies • actual diagnosis = actually faulty axioms Interactive Process: • repair of these axioms leads to intended ontology Given: multiple diagnoses Task: gather additional information to isolate actual diagnosis How? • ask a domain expert queries • use queries+answers as test cases to successively rule out diagnoses DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 3

Fault Localization – The Context e. g. : consistency, coherency required logiical properties Ontology Requirements Verifier required nonentailments required entailments all OK? Axioms Ontology actual diagnosis faulty axioms answer m delete axio ms ctify/is re. This the focus xio of a w ne ate l u form ana lyze DX‘ 19 query Interactive Fault Localization axio m Simple Expert Questioning Approach for Ontology Fault Localization Expert this work! st pl udy an e at xio n( s) possible diagnoses Interactive Repair 4

A Simple Example Ontology Requirements Verification Axioms Requirements A Sub. Class. Of B consistency B Sub. Class. Of C A Sub. Class. Of C x Type not C x Type A Ontology correct? not entails any axiom in? not violates any property in? Expert DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 5

A Simple Example Interactive Fault Localization two possible diagnoses • Axioms A Sub. Class. Of B consistency B Sub. Class. Of C A Sub. Class. Of C actual diagnosis x Type not C x Type A x Type B ? Interactive Fault Localization x Type B DX‘ 19 yes! Simple Expert Questioning Approach for Ontology Fault Localization Expert 6

Queries in Action! (Debugging with our Onto. Debug Tool) Ontology Queries Diagnoses DX‘ 19 Test Cases Simple Expert Questioning Approach for Ontology Fault Localization http: //isbi. aau. at/ontodebug 7

Fault Localization – The Goals 1. Finding the actual diagnosis (which allows to formulate the intended ontology) 2. Minimizing the effort for the expert 3. Minimizing the waiting time of the expert (= computation time of fault localization system) Goal 1: Locate faulty axioms Axioms Ontology actual diagnosis faulty axioms expensive!! Goal 3: Reduce / Optimize computations Interactive Fault Localization query minimize! answer Expert Goal 2: Minimize user interactions DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 8

Fault Localization – The Ingredients • • The way of interacting with the expert Assumptions about (behavior of) expert A criterion to be optimized Algorithm for query computation / selection General assumption: actual diagnosis faulty axioms Goal 3: Reduce / Optimize computations Interactive Fault Localization Ontology Query computation? DX‘ 19 to define a query? ) will expert answer queries? ) to measure expert‘s effort? ) to compute a query? ) Expert is knowledgable + able to classify axioms as entailments / non-entailments of the intended ontology Goal 1: Locate faulty axioms Axioms (how What to minimize? Simple Expert Questioning Approach for Ontology Fault Localization Definition? Answering strategy? query minimize! answer Expert Goal 2: Minimize user interactions 9

Fault Localization Ingredients Analyzed + Challenged What to minimize? Definition? Answering strategy? Query computation? • Interactive Fault Localization query minimize! answer Expert very expensive due to reasoning! do as rarely as possible! diagnoses Axioms Ontology DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 10

Fault Localization Ingredients Analyzed + Challenged • Query computation? assumption: each query is equally costly to answer What to minimize? query Interactive Fault Localization minimize! answer s po DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization Answering strategy? Definition? Expert ne g 11

Fault Localization Ingredients Analyzed + Challenged What to minimize? Query computation? • DX‘ 19 Definition? Simple Expert Questioning Approach for Ontology Fault Localization Interactive Fault Localization Answering strategy? query minimize! answer Expert 12

Fault Localization Ingredients Analyzed + Challenged What to minimize? Definition? Query computation? • Interactive Fault Localization Answering strategy? query minimize! answer Expert Each axiom-based answer is strictly more informative than query-based one Axiom-based strategy means better diagnoses elimination! DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 13

What to minimize? Fault Localization Query computation? Ingredients Analyzed + Challenged Interactive Fault Localization • s po DX‘ 19 Definition? ne Simple Expert Questioning Approach for Ontology Fault Localization g Answering strategy? query minimize! answer Expert binary evaluation! 14

Fault Localization Ingredients Analyzed + Challenged Query computation? • DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization What to minimize? Interactive Fault Localization Definition? Answering strategy? query minimize! answer Expert 15

Fault Localization Ingredients Analyzed + Challenged Summary (analysis of existing works): • efficiency depends on (behavior of) interacting expert • (query) optimizations might be inefficient or only approximations • optimization criteria might not be fully realistic Query computation? What to minimize? Interactive Fault Localization DX‘ 19 Definition? Answering strategy? query minimize! answer Expert Simple Expert Questioning Approach for Ontology Fault Localization 16

Our Solution Use singleton queries! Note: For singleton queries • there are exactly 2 outcomes • (necessarily) unique expert behavior (all expert types coincide) • #Ax = #Q DEF: A singleton query is a query which includes exactly one axiom. Singleton queries are simple + solve all problems we discussed! Advantages are in particular: Better informed computations: Each computed query-axiom depends on all so-far acquired expert inputs. Smaller search space: Worst-case costs to search for best singleton query are less than for best normal query. Realistic query evaluation (independent of given expert type): Binary (heuristic) evaluation, as done by existing works, is exact + plausible for singleton queries. Direct reuse of existing works: Concepts (e. g. , heuristics, UIs) for normal queries immediately reusable for singleton queries. Unequivocal optimization criterion: Two competing and arguable views (#Ax vs. #Q) on optimization problem are unified. DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 17

Our Solution • Note: For singleton queries • there are exactly 2 outcomes • (necessarily) unique expert behavior (all expert types coincide) • #Ax = #Q Note: (1) a set of diagnoses is given as input, (2) poly time + space holds independently of the used logic DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 18

Evaluation Answer: • for normal queries: • for singleton queries: yes no significant difference (for both #Ax and #Q) just one answering behavior! Question 1: Does the expert answering behavior make a difference (wrt. fault localization cost)? results for one of the tested ontologies – others very similar how many axioms must be classified? how many queries must be answered? expert types expert just says pos/neg to the query expert says pos/neg to query-axioms query selection heuristics DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 19

Evaluation Answer: • wrt. #Ax: pragmatist strategy is always superior (on avg) to all others • wrt. #Q: not clear-cut, but overall pragmatist tends to be best Recommend the expert to go through query-axioms + answer each single axiom until first negative one is found Question 2: How should the expert answer normal queries? Which strategy is the best? results for one of the tested ontologies – others very similar how many axioms must be classified? how many queries must be answered? expert types query selection heuristics DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 20

Evaluation Answer: (considering all experiments over all tested ontologies) • wrt. #Ax: singleton queries led to less expert effort in 66% of the cases • wrt. #Q : interestingly, singleton queries mostly the better choice as well Question 3: What is better, singleton or normal queries? • data for entropy heuristic • similar results for other heuristics if white dot (median) above red line, singleton queries better in majority of cases 20 (random) fault localization sessions per ontology DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 21

Evaluation Answer: • singleton queries • time savings against normal queries substantial • reason: smaller search space Question 4: Which approach (query type + answering strategy) is computationally most efficient? (wrt. the expert‘s waiting time between two queries) avg expert waiting time (ms) results for all tested ontologies times measured for computation of optimal queries (wrt. given query selection heuristic) normal queries: different expert types DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 22

Conclusions & Future Work Challenged existing interactive ontology fault localization methods wrt. – assumptions they make – criteria they optimize – interaction means they use Existing methods: – efficiency depends largely on behavior of interacting expert – (query) optimizations in general either inefficient or only approximations – optimization criteria (often) not fully realistic Suggested novel (and simpler!) interaction approach: – solves all problems of existing techniques – more efficient than existing methods • wrt. expert effort + computation time • independent of expert behavior • wrt. both (the conventional and a more realistic) optimization criteria Future work: – Find general algorithm for new approach, analyze further (expert) scenarios DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 23

Thank you for your attention! On the Usefulness of Different Expert Question Types for Fault Localization in Ontologies DX’ 19 Patrick Rodler, Michael Eichholzer University of Klagenfurt

Onto. Debug! Give it a try http: //isbi. aau. at/ontodebug DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 25

APPENDIX DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 26

Ontology Debugging Specifying Requirements / Focusing on Relevant Axioms • Search for faults only here! DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 27

Ontology Debugging Fault Detection Problem Axioms • not entails any axiom in not violates any property in DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 28

Ontology Debugging Fault Localization Problem (1) – Finding a Diagnosis Axioms • Note: In Onto. Debug diagnoses are called “repairs“ not entails any axiom in not violates any property in DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 29

Ontology Debugging Fault Identification and Repair Problem (1) • DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 30

Ontology Debugging Fault Identification and Repair Problem (2) • violates s violate DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 31

Example Fault Identification and Repair Axioms A Sub. Class. Of B x Type not C x Type B x Type A B Sub. Class. Of C A Sub. Class. Of C consistency Typo! Explanation 3 1 2 B Sub. Class. Of D DX‘ 19 Simple Expert Questioning Approach for Ontology Fault Localization 32