Qualitative Reverse Engineering Dorian uc and Ivan Bratko

Qualitative Reverse Engineering Dorian Šuc and Ivan Bratko Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia

Overview • Qualitative reverse engineering: – generating explanation of engineering designs – in qualitative terms • Case study: industrial crane controller – regression, model trees – qualitative trees • Discussion

Motivation Accumulated knowledge in companies: libraries of designs and corresponding simulation models • But: incomplete documentation – Why did designer move from one model to another? – Why was a design abandoned in favor of another? – How a designed system works? • Difficulties in re-use of design libraries: – intuitions behind designs are not explained

Context • Clockwork European project: – computer aided engineering design – industrial partners: car and heating industry – libraries of simulation models • Automated explanation generation – knowledge sharing: making tacit knowledge available to other people – model re-use: for model to be used by another designer, generate explanation how model works

The problem • Given designed system • Explain: – how the system works? – ideas, intuitions behind the design • Approach: – use ML on traces of system’s behavior – similar to behavioural cloning (Michie et al. , 90; Sammut et al. , 92)

Case studies • Reverse engineering of industrial crane controller, presented in this paper • Reverse engineering of car suspension system

Case study: anti-sway crane For transportation of slabs Designed by Czech Technical Univ. In daily use Crane parameters: travel 100 m height 15 m, width 30 m 80 -120 tons

Anti-sway crane Control force: F State: c. X, c. V, l. X, l. V ? Anti-sway controller desired carriage velocity c. VDes control force F • achieves c. VDes • minimizes swing

Anti-sway crane

Matlab model of crane controller

Learning problem controller = ? Given simulation model + controller explain how controller works

Learning problem

Learning problem c. VDes F c. X c. V l. X l. V 2. 0 0 0. 0 24914 0. 1 0. 3 2. 0 40188 0. 4 2. 1 2. 0 -425 1. 6 3. 1 1. 5 2. 8 … …. . … … ML

ML methods used • Linear regression • Regression, model trees • Qualitative trees (QUIN)

QUalitative INduction (QUIN) • Learning of qualitative trees • Qualitative tree defines a partition of the state space into areas with common qualitative behavior • Qualitatively constrained functions (QCFs) in leaves: – generalization of monotonicity constraint M+ – similar to qualitative proportionality predicates (QPT -Forbus, 84)

Example problem for QUIN • Noisy examples generated from z=x 2 -y 2

Qualitative patterns in data x > 0 & y > 0 => z = M+, -(x, y)

Induced qualitative tree for function z=x 2 -y 2 z is monotonically increasing in its dependence on x and monotonically decreasing in its dependence on y z is positively related to x and negatively related to y

QUIN algorithm • MDL to learn “most consistent” QCF • Learning qualitative tree (similar to ID 3) – for every possible split, split the examples into two subsets, find the “most consistent” QCF for both subsets and select the split minimizing tree-error cost (based on MDL) • QUIN is heuristic improvement of this idea

Related work in learning qualitative models • Previous work includes: – – – Mozetič, 87 (KARDIO), Coiera, 89; Hau, Coiera, 97 Bratko, Muggleton, Varšek, 91 Richards, Kraan, Kuipers, 92 Say, Kuru, 96 Kay, Rinner, Kuipers, 2000 • Most of these systems induce QDE models, hard to apply to our problem

Reverse engineering of anti-sway crane controller ML from examples of controller’s execution: • • Linear regression, regression and model trees (Weka-Witten&Frank, 2000; M 5 -Quinlan, 92) QUIN

Reverse engineering of anti-sway crane controller ML from examples of controller’s execution: • • Linear regression, regression and model trees (Weka-Witten&Frank, 2000; M 5 -Quinlan, 92) QUIN Given: • an execution trace • c. VDes=2 c. VDes=0 of anti-sway controller (follow c. VDes, minimize swing) learn control force F = F(l. X, l. V, c. VDes)

Evaluation criteria Comparing the induced models: • suitability as explanation • control performance – criterion of controller’s success: mean abs. velocity error and mean abs. swing error bellow errors of anti-sway controller + 10%

Regression and model trees Learning F=F (l. X, l. V, c. VDes): • • pruning: smallest non-single leaf tree has 34 leaves control performance: – cannot work: c. VDes does not appear in any of induced trees Helping the learning system: • • • derived attribute: relative carriage velocity c. VRel=c. V-c. VDes pruning: smallest non-single leaf tree has 17 leaves control performance: – does not move(F 0 when l. X 0), or huge swing Summary: • complex trees, not even close to achieving the task • not suitable as explanation

Linear regression Learning F=F (l. X, l. V, c. VDes): • F = 23218 c. VDes - 18578 c. V - 20319 l. X + 12410 l. V - 4800 • bad fit (R=0. 73, RRE=69. 12%) • unsatisfactory control performance Helping the learning system (using c. VRel): • Similar, even slightly worse results Summary: • simple models • but unsatisfactory control performance and bad fit

Experiments with QUIN Using c. VRel: accelerate yes no Small swing => follow c. VDes (since c. VRel=c. V-c. VDes) F=M-(l. X), l. X decreas. => increase F Large swing => reduce swing

Experiments with QUIN, cont. F = F(l. X, l. V, c. VDes): yes no Equivalent to previous qualitative tree! (given the particular crane dynamics)

Control with qualititative trees • By transforming QCFs in leaves into real-valued functions satisfying qualitative constraints M+ ( X ) f+(X) Randomly generated real -valued increasing functions Arbitrary increasing fn. of X • Testing 100 transformed qual. trees for control: – 43 successful (according to criterion of success)

Anti-sway clone

Reverse engineering of car suspension system • Given Quarter Car Model, reverse engineer controller for semi-active car suspension • Matlab/Simulink model provided by CTU and Intec

Car suspension model The original controller Black Box The shock absorber characteristics

Reverse engineering of car suspension system • Qualitative controller was induced with QUIN • Induced controller applied to control car model • Performance compared with original controller

Performance comparison Original controller: Qualitative controller: Comparatively better Comparatively worse

Summary of approach • Reverse engineering in qualitative terms, useful to grasp the intuition behind the design • Induction of qualitative trees (QUIN) • Qualitative-to-quantitative transformation of qualitative control strategies (involves optimisation)

Qualitative vs. quantitative models • Induced qualitative models superior as explanations (unsurprisingly) • Induced qualitative controllers* also perform better (surprisingly!) * “Qualitative controller” means: qualitative control strategy induced by QUIN, applied as controller after qualitative-to-quantitative transformation

Why numerical learning fails? 1. Linear regression alone insufficient easy to see 2. But, why model trees also fail? 3. Why qualitative trees succeed when model trees fail?

Example from crane • M 5 model tree contains leaf - 0. 0055 < l. X =< 0. 0055 Corresponding example set is very awkward for linear regression, see visualisation • QUIN finds better split: - 0. 223 < l. X =< 0. 163 This example set exhibits clearer qualitative pattern

Example from crane, cont.

Example from crane, cont.

Tentative explanation • QUIN can detect subtle qualitative patterns, although they are obscured by large quantitative changes • Regression tree learning (M 5) does not find good splits, resulting in regression-unfriendly leaves

Comparison Qualitative trees more successful: – suitability as explanation – control performance Why regression trees fail? • Important patterns overshadowed by numerically dominant attributes – c. VDes does not appear in regression trees! • Still qualitatively visible