Assigning Metrics for Optimization Evaluation Measures Each evaluation

Assigning Metrics for Optimization

Evaluation Measures • Each evaluation measure (EM) is a category by which an option is ranked/graded – Example: A car can be ranked by mileage, top speed, number of gears, and seating capacity. • A metric (or value function) must be established for each EM

Value Functions • A single value metric must be specified for each EM: – Sometimes, lower ratings are preferred • – Sometimes, higher ratings are preferred • – • Example: lower cost Example: higher mileage Choose the one that is applicable to the case. The scale of each metric must then be normalized to a range of 1.

Choosing a Value Function for an Evaluation Measure • There are two types of value functions: – Piecewise linear • Used for arbitrary scales of performance, attractiveness, etc. • Few data points are available • Good for expressing discontinuity – Exponential • Used for stress, deflection, and other physical or concrete factors. • Many data points can be obtained • Suitable for incorporating a risk factor

Example: Piecewise Linear Value Function • The x-axis shows the grading scale for productivity (chosen arbitrarily) • The y-axis shows the relative value of each grade • In this case, an improvement from a grade of -1 to 0 is as much as an improvement from 0 to 2.

Risk Tolerance • How daring is the decision maker? – Risk tolerant: report a better score than is calculated by the metric (positive r) – Risk averse: report a worse score than is calculated (negative r) – Risk neutral: report the actual score (r= ) • The risk level r can be – Calculated using the method in the next slide – Estimated by looking at the different graphs shown in the following slides or created by a simulation.

Technical Method for Solving ρ • • ρ > 0. 1 * Range of Measurement Find mid-value score (i. e. value = 0. 5) Assign value of 0. 5 to a specific score Solve numerically or use a table – Calculate normalized mid-value (range of scores 1) – Find normalized ρ – De-normalize ρ by multiplying by the range

Exponential Value Function • Higher scores are better • Risk tolerance r – r > 0: risk tolerant – r < 0: risk averse

Exponential Value Function (cont. ) • Lower scores are better

Final Evaluation • Combine value function grades with their respective weights to calculate the final grade/value

Bicycle Example • In our example, the EMs are: – Cost • Measured in “dollars” • Lower score is preferred – Weight • Measured in “pounds” • Lower score is preferred – Lifetime • Measured in “months” • Higher score is preferred

Bicycle Example (cont). • We use the exponential value function. – Use a risk-averse outlook for personal safety, e. g. set ρ = -5. • The exponential value function has been built into an MS Excel module and will be explained/utilized in later lectures.