CS 213 Floating Point 452006 Topics n n

CS 213 Floating Point 4/5/2006 Topics n n – 1– IEEE Floating Point Standard Rounding Floating Point Operations Mathematical properties CS 213, S’ 06

Floating Point Puzzles n For each of the following C expressions, either: l Argue that it is true for all argument values l Explain why not true • x == (int)(float) x int x = …; • x == (int)(double) x float f = …; • f == (float)(double) f double d = …; • d == (float) d • f == -(-f); Assume neither d nor f is Na. N • 2/3 == 2/3. 0 • d < 0. 0 ((d*2) < 0. 0) • d > f -f > -d • d * d >= 0. 0 • (d+f)-d == f – 2– CS 213, S’ 06

IEEE Floating Point Representations n Encodes rational numbers of the form V=x*(2^y) n Useful for very large numbers or numbers close to zero IEEE Standard 754 n Established in 1985 as uniform standard for floating point arithmetic l Before that, many idiosyncratic formats n Supported by all major CPUs Driven by Numerical Concerns n n Nice standards for rounding, overflow, underflow Hard to make go fast l Numerical analysts predominated over hardware types in defining standard – 3– CS 213, S’ 06

Fractional Binary Numbers 2 i 2 i– 1 4 2 1 bi bi– 1 • • • b 2 b 1 b 0. b– 1 b– 2 b– 3 1/2 1/4 1/8 • • • b–j • • • 2–j Representation – 4– n Bits to right of “binary point” represent fractional powers of 2 n Represents rational number: CS 213, S’ 06

Frac. Binary Number Examples Value Representation 5 -3/4 2 -7/8 63/64 101. 112 10. 1112 0. 1111112 Observations Divide by 2 by shifting right (the point moves to the left) n Multiply by 2 by shifting left (the point moves to the right) n Numbers of form 0. 111111… 2 just below 1. 0 n l 1/2 + 1/4 + 1/8 + … + 1/2 i + … 1. 0 l Use notation 1. 0 – – 5– CS 213, S’ 06

Representable Numbers Limitation Can only exactly represent numbers of the form x/2 k n Other numbers have repeating bit representations n Value Representation 1/3 0. 010101[01]… 2 1/5 0. 00110011[0011]… 2 1/10 0. 000110011[0011]… 2 – 6– CS 213, S’ 06

Floating Point Representation Numerical Form n – 1 s M 2 E l Sign bit s determines whether number is negative or positive l Significand M normally a fractional value in range [1. 0, 2. 0). l Exponent E weights value by power of two Encoding s exp frac MSB is sign bit n exp field encodes E n frac field encodes M n – 7– CS 213, S’ 06

Floating Point Precisions Encoding s exp frac MSB is sign bit n exp field encodes E n frac field encodes M n Sizes n Single precision: 8 exp bits, 23 frac bits l 32 bits total n Double precision: 11 exp bits, 52 frac bits l 64 bits total n Extended precision: 15 exp bits, 63 frac bits l Only found in Intel-compatible machines l Stored in 80 bits » 1 bit wasted – 8– CS 213, S’ 06

“Normalized” Numeric Values Condition n exp 000… 0 and exp 111… 1 Exponent coded as biased value E = Exp – Bias l Exp : unsigned value denoted by exp l Bias : Bias value » Single precision: 127 (Exp: 1… 254, E: -126… 127) » Double precision: 1023 (Exp: 1… 2046, E: -1022… 1023) » in general: Bias = 2 e-1 - 1, where e is number of exponent bits Significand coded with implied leading 1 M = 1. xxx…x 2 l xxx…x: bits of frac l Minimum when 000… 0 (M = 1. 0) l Maximum when 111… 1 (M = 2. 0 – ) l Get extra leading bit for “free” – 9– CS 213, S’ 06

Normalized Encoding Example Value Float F = 15213. 0; n 1521310 = 111011012 = 1. 11011012 X 213 Significand M = frac = 1. 11011012 1101101000002 Exponent E = Bias = Exp = 13 127 140 = 100011002 Floating Point Representation: Hex: Binary: 140: 15213: – 10 – 4 6 6 D B 4 0 0 0100 0110 1101 1011 0100 0000 100 0110 0 1101 1011 01 CS 213, S’ 06

Denormalized Values Condition n exp = 000… 0 Value n Exponent value E = –Bias + 1 l Note: not simply E= – Bias n Significand value M = 0. xxx…x 2 l xxx…x: bits of frac Cases n exp = 000… 0, frac = 000… 0 l Represents value 0 l Note that have distinct values +0 and – 0 n exp = 000… 0, frac 000… 0 l Numbers very close to 0. 0 – 11 – CS 213, S’ 06

Special Values Condition n exp = 111… 1 Cases n exp = 111… 1, frac = 000… 0 l Represents value (infinity) l Operation that overflows l Both positive and negative l E. g. , 1. 0/0. 0 = 1. 0/ 0. 0 = + , 1. 0/ 0. 0 = n exp = 111… 1, frac 000… 0 l Not-a-Number (Na. N) l Represents case when no numeric value can be determined l E. g. , sqrt(– 1), – 12 – CS 213, S’ 06

Summary of Floating Point Real Number Encodings Na. N – 13 – -Normalized +Denorm -Denorm 0 +0 +Normalized + Na. N CS 213, S’ 06

Tiny Floating Point Example 8 -bit Floating Point Representation n the sign bit is in the most significant bit. n the next four bits are the exponent, with a bias of 7. the last three bits are the frac n l Same General Form as IEEE Format n n normalized, denormalized representation of 0, Na. N, infinity 7 6 s – 14 – 0 3 2 exp frac CS 213, S’ 06

Values Related to the Exponent – 15 – Exp exp E 2 E 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 -6 -6 -5 -4 -3 -2 -1 0 +1 +2 +3 +4 +5 +6 +7 n/a 1/64 1/32 1/16 1/8 1/4 1/2 1 2 4 8 16 32 64 128 (denorms) (inf, Na. N) CS 213, S’ 06

Dynamic Range E Value 0000 001 0000 010 -6 -6 -6 0 1/8*1/64 = 1/512 2/8*1/64 = 2/512 closest to zero 0000 0001 110 111 000 001 -6 -6 6/8*1/64 7/8*1/64 8/8*1/64 9/8*1/64 = = 6/512 7/512 8/512 9/512 largest denorm smallest norm 0110 0111 110 111 000 001 010 -1 -1 0 0 0 14/8*1/2 15/8*1/2 8/8*1 9/8*1 10/8*1 = = = 14/16 15/16 1 9/8 10/8 7 7 n/a 14/8*128 = 224 15/8*128 = 240 inf s exp 0 0 Denormalized 0 … numbers 0 0 … 0 0 Normalized 0 numbers 0 0 … 0 0 0 – 16 – frac 1110 1111 000 closest to 1 below closest to 1 above largest norm CS 213, S’ 06

Distribution of Values 6 -bit IEEE-like format n e = 3 exponent bits n f = 2 fraction bits Bias is 3 n Notice how the distribution gets denser toward zero. – 17 – CS 213, S’ 06

Distribution of Values (close-up view) 6 -bit IEEE-like format n e = 3 exponent bits n f = 2 fraction bits Bias is 3 n Note: Smooth transition between normalized and denormalized numbers due to definition E = 1 - Bias for denormalized values – 18 – CS 213, S’ 06

Interesting Numbers Description Zero exp frac Numeric Value 00… 00 0. 0 Smallest Pos. Denorm. 00… 00 00… 01 2– {23, 52} X 2– {126, 1022} Single 1. 4 X 10– 45 n Double 4. 9 X 10– 324 n Largest Denormalized 00… 00 11… 11 (1. 0 – ) X 2– {126, 1022} Single 1. 18 X 10– 38 n Double 2. 2 X 10– 308 n Smallest Pos. Normalized 00… 01 00… 00 1. 0 X 2– {126, 1022} n One Just larger than largest denormalized 01… 11 00… 00 1. 0 Largest Normalized 11… 10 11… 11 (2. 0 – ) X 2{127, 1023} Single 3. 4 X 1038 n Double 1. 8 X 10308 n – 19 – CS 213, S’ 06

Floating Point Operations Conceptual View n First compute exact result n Make it fit into desired precision l Possibly overflow if exponent too large l Possibly round to fit into frac Rounding Modes (illustrate with $ rounding) $1. 40 $1. 60 $1. 50 Zero $1 $1 n Round down (- ) $1 n Round up (+ ) $2 n Nearest Even (default) n $2. 50 –$1. 50 $1 $1 $2 $2 –$1 $2 $3 $2 –$1 $2 –$2 Note: 1. Round down: rounded result is close to but no greater than true result. 2. Round up: rounded result is close to but no less than true result. – 20 – CS 213, S’ 06

Closer Look at Round-To-Even Default Rounding Mode n All others are statistically biased l Sum of set of positive numbers will consistently be over- or under- estimated Applying to Other Decimal Places / Bit Positions n When exactly halfway between two possible values l Round so that least significant digit is even n E. g. , round to nearest hundredth 1. 2349999 1. 2350001 1. 2350000 1. 2450000 – 21 – 1. 23 1. 24 (Less than half way) (Greater than half way) (Half way—round up) (Half way—round down) CS 213, S’ 06

Rounding Binary Numbers Binary Fractional Numbers “Even” when least significant bit is 0 n Half way when bits to right of rounding position = 100… 2 n Examples Round to nearest 1/4 (2 bits right of binary point) Value Binary Rounded Action Rounded Value 2 3/32 10. 000112 10. 002 (<1/2—down) 2 2 3/16 10. 001102 10. 012 (>1/2—up) 2 1/4 2 7/810. 111002 11. 002 (1/2—up) 3 2 5/810. 101002 10. 102 (1/2—down) 2 1/2 n – 22 – CS 213, S’ 06

FP Multiplication Operands (– 1)s 1 M 1 2 E 1 * (– 1)s 2 M 2 2 E 2 Exact Result (– 1)s M 2 E n n n Sign s: s 1 ^ s 2 Significand M: M 1 * M 2 Exponent E: E 1 + E 2 Fixing n If M ≥ 2, shift M right, increment E n If E out of range, overflow Round M to fit frac precision n Implementation n – 23 – Biggest chore is multiplying significands CS 213, S’ 06

FP Addition Operands (– 1)s 1 M 1 2 E 1 (– 1)s 2 M 2 2 E 2 n E 1–E 2 (– 1)s 1 M 1 Assume E 1 > E 2 Exact Result (– 1)s M 2 E n (– 1)s 2 M 2 + Sign s, significand M: l Result of signed align & add n Exponent E: E 1 Fixing n If M ≥ 2, shift M right, increment E n if M < 1, shift M left k positions, decrement E by k Overflow if E out of range Round M to fit frac precision n – 24 – n CS 213, S’ 06

Mathematical Properties of FP Add Compare to those of Abelian Group n Closed under addition? YES l But may generate infinity or Na. N Commutative? n Associative? n YES NO l Overflow and inexactness of rounding » (3. 14+1 e 10)-1 e 10=0 (rounding) » 3. 14+(1 e 10 -1 e 10)=3. 14 0 is additive identity? YES n Every element has additive inverse n ALMOST l Except for infinities & Na. Ns Monotonicity n a ≥ b a+c ≥ b+c? ALMOST l Except for Na. Ns – 25 – CS 213, S’ 06

Math. Properties of FP Mult Compare to Commutative Ring n Closed under multiplication? YES l But may generate infinity or Na. N Multiplication Commutative? n Multiplication is Associative? n YES NO l Possibility of overflow, inexactness of rounding 1 is multiplicative identity? YES n Multiplication distributes over addition? NO n l Possibility of overflow, inexactness of rounding Monotonicity n a ≥ b & c ≥ 0 a *c ≥ b *c? ALMOST l Except for Na. Ns – 26 – CS 213, S’ 06

Floating Point in C C Guarantees Two Levels float double single precision double precision Conversions n n Casting between int, float, and double changes numeric values Double or float to int l Truncates fractional part l Like rounding toward zero l Not defined when out of range » Generally saturates to TMin or TMax n int to double l Exact conversion, as long as int has ≤ 53 bit word size n int to float l Will round according to rounding mode – 27 – CS 213, S’ 06

Summary IEEE Floating Point Has Clear Mathematical Properties n Represents numbers of form M X 2 E n Not the same as real arithmetic l Violates associativity/distributivity l Makes life difficult for compilers & serious numerical applications programmers – 28 – CS 213, S’ 06