Aristotle University of Thessaloniki Greece Department of Informatics

Presentation structure The evaluation problem Similar cases identification Merging of past cases Example Conclusions

The evaluation problem n An evaluation problem P can be modeled as a 7

The ISO-9126 software evaluation model 4 SETN-02 – 2 nd Hellenic Conference on Artificial

Attributes n Attribute classification n Basic attributes n n Compound attributes Types of evaluation

Multi-criteria aggregation methods n Utility-based methods n n Outranking methods n n 6 Assess

Our motivation n n Having a knowledge base of past evaluation models, combine these

Problem descriptors (1/2) n n n 9 Each evaluation case, either past or new,

Problem descriptors (2/2) n n 10 For simplicity, we require that all distances range

Distance between problems n For an evaluation problem P, V(P) denotes the Ndimensional vector

Success and Value of a Model n n Each past model MDi is characterized

Model selection n Select past models using one of the following strategies: n n

Defining the new model (1/2) n n n 15 Based on the selected past

Defining the new model (2/2) n n 16 An attribute that does not appear

Weights and scales for arithmetic attributes 17 SETN-02 – 2 nd Hellenic Conference on

Creating nominal scales n n Let Na, i denote the nominal scale of attribute

Evaluation problem concerning the choice of 1 out-of-n commercial ERP systems. Descriptor System users

Selection of past models n n wprox=0. 3, wsucc=0. 7 Two selected models: n

Creating the new model 22 # Attribute MD 1 MD 2 MD MD’ 1

Conclusions n Synthesis of evaluation models based on past cases. n n n Future

Slides: 25

Download presentation

Aristotle University of Thessaloniki, Greece Department of Informatics Decision Making Based on Past Problem Cases Ioannis Stamelos and Ioannis Refanidis {stamelos, yrefanid}@csd. auth. gr SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Presentation structure The evaluation problem Similar cases identification Merging of past cases Example Conclusions SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

The evaluation problem n An evaluation problem P can be modeled as a 7 -ple MD = (A, T, D, M, E, G, R) where: n n n n 3 A is the set of alternatives under evaluation. T is the type of the evaluation. D is the tree of the evaluation attributes. M is the set of the associated measures. E is the set of scales associated to the attributes. G is a set of rules representing the user's preferences. R is the preference aggregation procedure. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

The ISO-9126 software evaluation model 4 SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Attributes n Attribute classification n Basic attributes n n Compound attributes Types of evaluation models n n 5 Arithmetic attributes Nominal attributes Fixed models Constructive models SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Multi-criteria aggregation methods n Utility-based methods n n Outranking methods n n 6 Assess the basic attributes for each alternative and use weighted average sum to compute an overall value for each alternative Compare pairs of alternatives Alternative A outranks alternative B for a compound criterion g, if the percentage of g’s descendents for which A outranks B (taking into account their weights) is greater than a user-specified threshold. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Our motivation n n Having a knowledge base of past evaluation models, combine these models into a new one. Factors that must be considered: n n 7 The proximity between the past problems and the new one. The success of the application of the past models. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Problem descriptors (1/2) n n n 9 Each evaluation case, either past or new, is characterized with a set of descriptors, d 1, d 2, . . . , d. M, each one of them being accompanied by a weight wi. To each descriptor di a set of (possible) values Vi has also to be assigned. For each pair of values vi, x, vi, y, Vi, their distance dist(vi, x, vi, y) has to be defined. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Problem descriptors (2/2) n n 10 For simplicity, we require that all distances range in the interval [0, 1], where a zero distance denotes identical problems with respect to a specific dimension. In the simplest case, Vi=[0, 1], and the distance between any two values vi, x, vi, y Vi is defined as dist(vi, x, vi, y)=|vi, x-vi, y| ranging in the interval [0, 1]. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Distance between problems n For an evaluation problem P, V(P) denotes the Ndimensional vector with the values of P in the descriptors, whereas Vi(P) denotes the value of its i -th dimension. n Proximity: prox(P 1, P 2)=1 -dist(P 1, P 2) 11 SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Success and Value of a Model n n Each past model MDi is characterized with a degree of success, succ(MDi) We assign preferences to the proximity and the success of past evaluation models, in the form of weights: n n The value of a past model MDi with respect to the new problem P is defined as: n 12 wprox+wsucc=1 Value(MDi)=wprox(Pi, P) + wsucc(MDi) SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Model selection n Select past models using one of the following strategies: n n n 13 A constant number of the past evaluation models, with greater values in Value. All the past models, whose value Value is greater than a threshold Valuemin. A specific percentage of the models in the knowledge base, which have the greatest values in Value. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Defining the new model (1/2) n n n 15 Based on the selected past evaluation models, we have to define the new model. We suppose a universal attribute hierarchy structure, i. e. all models have the same attributes in the same places The models differ only in the weights that have been assigned to the attributes and in the scales of the basic ones. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Defining the new model (2/2) n n 16 An attribute that does not appear in an evaluation model MDi can be thought as having a zero weight. Under this point of view, the problem of determining the evaluation model MD is transformed to the problem of determining the weights of the attributes and the scales of the basic attributes for MD. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Weights and scales for arithmetic attributes 17 SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Creating nominal scales n n Let Na, i denote the nominal scale of attribute A in model MDi. Let aj, i denote whether value aj appears in model MDi, i. e. : n n n 18 aj, i=1, iff aj Na, i aj, i=0, iff aj Na, i Na={aj Ua: f(aj) tnominal} SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Evaluation problem concerning the choice of 1 out-of-n commercial ERP systems. Descriptor System users Weight [Scale] 0. 20 [7, 2000] Descriptor Weight [Scale] EDI interfaces 0. 05 [0, 10] Installation sites 0. 05 [0, 98] Conversions needed 0. 05 [1, 93] Plants involved 0. 10 [0, 98] Modifications needed 0. 10 [0, 30] Companies involved 0. 10 [1, 35] Requested reports 0. 05 [0, 100] User interfaces 0. 10 [0, 50] ERP modules acquired 20 0. 20 [1, 8] SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Selection of past models n n wprox=0. 3, wsucc=0. 7 Two selected models: n n 21 MD 1 such that Value(MD 1)=0. 9 MD 2 such that Value(MD 2)=0. 7 SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Creating the new model 22 # Attribute MD 1 MD 2 MD MD’ 1 Functionality 0. 15 0. 20 0. 17 0. 18 2 Reliability 0. 15 0. 10 0. 13 3 Usability 0. 10 4 Efficiency 0. 20 0. 16 5 Maintainability 0. 10 6 Portability 0. 00 0. 10 0. 04 0. 00 7 Cost 0. 30 0. 31 SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002

Conclusions n Synthesis of evaluation models based on past cases. n n n Future extensions n n 24 Proximity of the problems. The successful application of the past models. Computing statistical metrics of the new model. Integration with a friendly user interface. SETN-02 – 2 nd Hellenic Conference on Artificial Intelligence Thessaloniki, April 11 -12, 2002