Performance profiling of Experiments Geant 4 Simulations Geant

Performance profiling of Experiments’ Geant 4 Simulations Geant 4 Collaboration Workshop 14 th October, Lisbon Ryszard Jurga

Introduction to Performance Profiling § Goal § analysis of the behavior of a program while running • e. g. execution time, time spent per function, call graph § Software Instrumentation § code snippets to collect required data • source (manual, gcc -pg) • binary (offline, online) û high overhead § Hardware approach § special on-chip hardware of modern CPU • direct access to CPU resources (number of cycles, integer and floating point, instructions, branch prediction and missprediction, cache misses etc • event detectors, counters, Xeon (44, 18) ü less overhead § Hybrid solution § both: instrumentation and hardware solutions Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 2

openlab Performance Monitoring Stack Tools pfmon, perfex, Perf. Suite, q-tools, oprofile, caliper, gpfmon Libraries libperfctr, libpfm, PAPI Interfaces perfctr, perfmon Hardware x 86, IPF Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 3

Measurements § Measurements on Xeon counting/sampling/profiling ü user/kernel domain ü per process/system-wide ü counter multiplexing ü kernel 2. 4 & 2. 6 û number of function calls ü ü we work on it, will be available soon û call graph ü collaboration with interface developer in order to take advantage of hardware support Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 4

Tools - Perf. Suite § http: //perfsuite. ncsa. uiuc. edu (psrun, psprocess. . . ) § Open source collection of tools, utilities and libraries for software performance analysis ü basic counting ü sampling ü user metrics û no system-wide mode ü user/kernel domain ü 2. 4 & 2. 6 kernel ü profiling support û flat profile – neither number of function calls nor call graph Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 5

Perf. Suite - challenges û Does not work with python scripts running from command line û unpredictable behavior on AFS ( Andrew File System) û a problem with resolving function names unknown functions with static libraries û a huge problem with shared libraries û • in order to monitor successfully, Perf. Suite has to know all of them in advance – LD_PRELOAD variable – a big challange - how to select interesting libraries from 400+ without causing dependence error? – use other tools to have another look Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 6

Perf. Suite & LD_PRELOAD Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 7

Tools - gpfmon û there is no tool which meets our requirements in the domain of system-wide monitoring (counting, sampling) § we decided to develop our own tool § gpfmon • uses perfctr interface and library • user/kernel domain • per single or total CPU ü enables multiplexing • 4 even sets • cpu cycles, instructions completed, branches taken predicted and mispredicted, L 2 load and store missed, FP, scalar, load and stores instructions • we miss average 2% samples, apart from L 2 store missed – 92% Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 8

gpfmon sample results § Geant 4 Atlas simulation IPC - 0. 34 § FP - 18% § LD+ST - 63% (7% LD caused L 2 cache miss) § Branches - 10%, ratio taken predicted/mispredicted=36 § Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 9

Profiling § Atlas simulation • full events (30), minimum-bias (100) § LHCb simulation • 1000 events § Geant 4 examples • Test. Em 3 • example 04 • calorimeter regression test § Xeon 32 bit mode Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 10

Profiling results § Simulations Atlas § LHCb § § Geant 4 example N 04 Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 11

Libraries summary Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 12

Geant 4 libs Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 13

CLHEP libs Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 14

Profiling results - Atlas § Atlas simulations Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 15

Atlas@Xeon Libraries Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 16

Atlas@Xeon geant 4 libs Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 17

Atlas@Xeon functions Function Summary ----------------------------------------Samples Self % Total % Function 30728046 7320624 6689448 5873290 5490913 5278498 5050076 4919562 4503773 4366559 4295632 4199824 4033883 3938509 3912491 3766114 3722683 3620741 3620692 3604007 3598039 3581444 3259393 3127408 2872853 2833732 2737712 2733244 2721568 2654097 2610649 7. 82% 1. 86% 1. 70% 1. 49% 1. 40% 1. 34% 1. 28% 1. 25% 1. 11% 1. 09% 1. 07% 1. 03% 1. 00% 0. 96% 0. 95% 0. 92% 0. 91% 0. 83% 0. 80% 0. 73% 0. 72% 0. 70% 0. 69% 0. 68% 0. 66% 7. 82% ? ? 9. 68% G 4 Transportation: : Along. Step. Get. Physical. Interaction. Length() 11. 38% G 4 Voxel. Navigation: : Compute. Step() 12. 87% G 4 Navigator: : Compute. Step() 14. 27% G 4 Polycone. Side: : Intersect() 15. 61% G 4 Stepping. Manager: : Stepping() 16. 90% G 4 Propagator. In. Field: : Compute. Step() 18. 15% G 4 Navigator: : Locate. Global. Point. And. Setup() 19. 30% G 4 Polycone. Side: : Distance. Away() 20. 41% G 4 Intersecting. Cone: : Line. Hits. Cone 1() 21. 50% G 4 Voxel. Navigation: : Locate. Next. Voxel() 22. 57% G 4 Stepping. Manager: : Define. Physical. Step. Length() 23. 59% G 4 Multiple. Scattering 52: : Get. Continuous. Step. Limit() 24. 60% G 4 Stepping. Manager: : Invoke. PSDIP() 25. 59% G 4 Multiple. Scattering 52: : Post. Step. Do. It() 26. 55% G 4 Classical. RK 4: : Dumb. Stepper() 27. 50% G 4 Stepping. Manager: : Invoke. Along. Step. Do. It. Procs() 28. 42% _int_malloc 29. 34% G 4 Navigator: : Locate. Global. Point. And. Update. Touchable. Handle() 30. 25% LAr. Wheel. Calculator: : Distance. To. The. Neutral. Fibre() 31. 17% vfprintf 32. 08% G 4 Universal. Fluctuation: : Sample. Fluctuations() 32. 91% G 4 Electro. Nuclear. Cross. Section: : Get. Cross. Section() 33. 71% G 4 Polycone. Side: : Point. On. Cone() 34. 44% G 4 Navigation. Level. Rep: : G 4 Navigation. Level. Rep() 35. 16% G 4 Transportation: : Post. Step. Do. It() 35. 85% G 4 Chord. Finder: : Find. Next. Chord() 36. 55% G 4 Tubs: : Distance. To. In() 37. 24% G 4 VEnergy. Loss. Process: : Along. Step. Do. It() 37. 92% G 4 Mag. Error. Stepper: : Stepper() 38. 58% G 4 Particle. Change. For. Transport: : Update. Step. For. Along. Step() Ryszard Jurga, CERN openlab 2550219 2540628 2413526 2407946 2396557 2390632 2343439 2277101 2244256 2242211 2228671 2171964 2132567 2083184 2067235 1978418 1944091 1892005 1889364 1875195 1853968 1842997 1841307 1805522 1793612 1772012 1731569 1631892 1612075 1546908 1504737 1471951 1456046 1449124 0. 65% 0. 61% 0. 60% 0. 58% 0. 57% 0. 55% 0. 54% 0. 53% 0. 50% 0. 49% 0. 48% 0. 47% 0. 46% 0. 45% 0. 44% 0. 42% 0. 41% 0. 39% 0. 38% 0. 37% 39. 23% 39. 88% 40. 49% 41. 10% 41. 71% 42. 32% 42. 92% 43. 50% 44. 07% 44. 64% 45. 20% 45. 76% 46. 30% 46. 83% 47. 35% 47. 86% 48. 35% 48. 83% 49. 31% 49. 79% 50. 26% 50. 73% 51. 20% 51. 66% 52. 12% 52. 57% 53. 01% 53. 42% 53. 83% 54. 23% 54. 61% 54. 98% 55. 35% 55. 72% Geant 4 Workshop, Oct. 2006 G 4 Sandia. Table: : Get. Sandia. Cof. Per. Atom() G 4 Photo. Nuclear. Cross. Section: : Get. Cross. Section() G 4 Transportation: : Along. Step. Do. It() CLHEP: : Hep. James. Random: : flat() G 4 Poly. Phi. Face: : Intersect() G 4 Hadron. Cross. Sections: : Calc. Scattering. Cross. Sections() G 4 Mag. Int_Driver: : Quick. Advance() CLHEP: : Hep. Rotation: : rotate. Axes() G 4 Physics. Vector: : Get. Value() G 4 Stepping. Manager: : Set. Initial. Step() G 4 Tubs: : Inside() G 4 Normal. Navigation: : Compute. Step() G 4 VEm. Process: : Get. Mean. Free. Path() G 4 Voxel. Navigation: : Level. Locate() G 4 Voxel. Navigation: : Compute. Voxel. Safety() G 4 Voxel. Navigation: : Voxel. Locate() G 4 Propagator. In. Field: : Intersect. Chord() G 4 e. Bremsstrahlung. Model: : Sample. Secondaries() G 4 Polycone. Side: : Inside() G 4 Polycone. Side: : Distance() G 4 Affine. Transform: : G 4 Affine. Transform() G 4 Enclosing. Cylinder: : Must. Be. Outside() G 4 Step. Point: : operator=() G 4 Particle. Change: : Check. It() G 4 VCSGfaceted: : Distance. To. Out() G 4 Tracking. Manager: : Process. One. Track() G 4 Hadronic. Process: : Get. Mean. Free. Path() G 4 Mag. Int_Driver: : Accurate. Advance() G 4 Chord. Finder: : Advance. Chord. Limited() std: : vector<G 4 VTrajectory. Point*, std: : allocator<G 4 VTrajectory. Point*> >: : _M_insert_aux() G 4 VContinuous. Discrete. Process: : Post. Step. Get. Physical. Interaction. Length() G 4 VEnergy. Loss. Process: : Get. Mean. Free. Path() _int_free G 4 VCSGfaceted: : Distance. To. In() 18

Profiling results - LHCb § LHCb simulation Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 19

LHCb@Xeon libraries Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 20

LHCb@Xeon geant 4 libs Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 21

LHCb@Xeon functions Function Summary ----------------------------------------Samples Self % Total % Function 18672274 4. 49% ? ? 8998975 2. 16% 6. 66% G 4 Voxel. Navigation: : Compute. Step() 8123719 1. 95% 8. 61% G 4 Transportation: : Along. Step. Get. Physical. Interaction. Length() 7066865 1. 70% 10. 31% G 4 Voxel. Navigation: : Locate. Next. Voxel() 6757801 1. 63% 11. 93% G 4 Navigator: : Locate. Global. Point. And. Setup() 6716299 1. 62% 13. 55% G 4 Stepping. Manager: : Invoke. PSDIP() 6648670 1. 60% 15. 15% G 4 Navigator: : Compute. Step() 6237844 1. 50% 16. 65% G 4 Navigator: : Locate. Global. Point. And. Update. Touchable. Handle() 5054291 1. 22% 17. 86% G 4 Navigation. Level. Rep: : G 4 Navigation. Level. Rep() 5053875 1. 22% 19. 08% G 4 Universal. Fluctuation: : Sample. Fluctuations() 5015604 1. 21% 20. 29% CLHEP: : Hep. Rotation: : rotate. Axes() 4970036 1. 20% 21. 48% G 4 Stepping. Manager: : Invoke. Along. Step. Do. It. Procs() 4894483 1. 18% 22. 66% G 4 Stepping. Manager: : Stepping() 4785378 1. 15% 23. 81% G 4 Hadron. Cross. Sections: : Calc. Scattering. Cross. Sections() 4564152 1. 10% 24. 91% G 4 Box: : Distance. To. In() 4262441 1. 03% 25. 93% G 4 Voxel. Navigation: : Level. Locate() 3925503 0. 94% 26. 88% _int_malloc 3818722 0. 92% 27. 80% G 4 Transportation: : Along. Step. Do. It() 3662972 0. 88% 28. 68% G 4 Stepping. Manager: : Set. Initial. Step() 3480389 0. 84% 29. 51% G 4 Transportation: : Post. Step. Do. It() 3363825 0. 81% 30. 32% G 4 Msc. Model: : Sample. Cosine. Theta() 3363389 0. 81% 31. 13% G 4 Polycone. Side: : Distance. Away() 3343872 0. 80% 31. 94% G 4 VEnergy. Loss. Process: : Along. Step. Do. It() 3303118 0. 79% 32. 73% G 4 Stepping. Manager: : Define. Physical. Step. Length() 3232859 0. 78% 33. 51% G 4 Particle. Change: : Update. Step. For. Along. Step() 3201266 0. 77% 34. 28% G 4 e. Bremsstrahlung. Model: : Sample. Secondaries() 3198988 0. 77% 35. 05% G 4 Voxel. Navigation: : Compute. Voxel. Safety() 3081196 0. 74% 35. 79% CLHEP: : Ranlux. Engine: : flat() 2933359 0. 71% 36. 49% G 4 Material. Properties. Table: : Get. Property() Ryszard Jurga, CERN openlab 2918537 2881473 2879699 2708685 2708569 2702742 2642777 2570738 2533828 2404980 2380731 2375239 2364408 2192185 2173750 2138457 2101661 2073579 2070889 2023616 1982965 1982311 1925370 1921372 1909913 1843361 1828376 1804773 1801818 1752402 1730562 1693222 1670901 1648375 1630064 0. 70% 0. 69% 0. 65% 0. 64% 0. 62% 0. 61% 0. 58% 0. 57% 0. 53% 0. 52% 0. 51% 0. 50% 0. 49% 0. 48% 0. 46% 0. 44% 0. 43% 0. 42% 0. 41% 0. 40% 0. 39% 37. 20% 37. 89% 38. 58% 39. 23% 39. 88% 40. 53% 41. 17% 41. 79% 42. 40% 42. 98% 43. 55% 44. 12% 44. 69% 45. 22% 45. 74% 46. 25% 46. 76% 47. 26% 47. 76% 48. 24% 48. 72% 49. 20% 49. 66% 50. 12% 50. 58% 51. 02% 51. 46% 51. 90% 52. 33% 52. 75% 53. 17% 53. 58% 53. 98% 54. 37% 54. 77% Geant 4 Workshop, Oct. 2006 G 4 Polycone. Side: : Intersect() G 4 Tubs: : Inside() G 4 Material. Property. Vector: : Get. Property() G 4 Voxel. Navigation: : Voxel. Locate() G 4 Step. Point: : operator=() G 4 Box: : Inside() G 4 Physics. Vector: : Get. Value() G 4 Particle. Change. For. Transport: : Update. Step. For. Along. Step() G 4 Tubs: : Distance. To. In() G 4 Msc. Model: : Geom. Path. Length() G 4 Mag. Error. Stepper: : Stepper() G 4 Track: : Get. Velocity() G 4 VEm. Process: : Get. Mean. Free. Path() G 4 Intersecting. Cone: : Line. Hits. Cone 2() G 4 Mag. Int_Driver: : Accurate. Advance() G 4 Box: : Distance. To. Out() G 4 Sandia. Table: : Get. Sandia. Cof. Per. Atom() G 4 Particle. Change: : Initialize() G 4 Classical. RK 4: : Dumb. Stepper() G 4 Propagator. In. Field: : Compute. Step() G 4 Mag_Usual. Eq. Rhs: : Evaluate. Rhs. Given. B() G 4 Polycone. Side: : Distance() G 4 Box: : Distance. To. Out() G 4 Polycone. Side: : Inside() std: : _Rb_tree< >: : lower_bound() G 4 Cons: : Distance. To. In() G 4 Trd: : Inside() G 4 VDiscrete. Process: : Post. Step. Get. Physical. Interaction. Length() _int_free G 4 Affine. Transform: : G 4 Affine. Transform() G 4 VContinuous. Discrete. Process: : Post. Step. Get. Physical. Interaction. Length() G 4 Displaced. Solid: : Inside() G 4 Sphere: : Distance. To. In() G 4 Msc. Model: : Sample. Secondaries() G 4 VEnergy. Loss. Process: : Post. Step. Do. It() 22

Calorimeter Regression Test @Xeon § Geant 4. 8. 0. p 01 § “configuration which reproduces a typical LHC calorimeter” § atlas. HEC. g 4 : 5000 events of 100 Ge. V pi§ ATLAS HEC calorimeter (Copper-Liquid. Argon) § Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 23

CRT@Xeon libraries Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 24

CRT@Xeon geant 4 libs Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 25

CRT@Xeon functions Function Summary ----------------------------------------Samples Self % Total % Function 43663318 6. 07% ? ? 14323092 1. 99% 8. 06% G 4 e. Bremsstrahlung. Model: : Sample. Secondaries() 13989279 1. 94% 10. 00% G 4 Universal. Fluctuation: : Sample. Fluctuations() 10520122 1. 46% 11. 46% G 4 Msc. Model: : Sample. Cosine. Theta() 10485807 1. 46% 12. 92% G 4 Navigator: : Locate. Global. Point. And. Setup() 10151605 1. 41% 14. 33% G 4 Transportation: : Along. Step. Get. Physical. Interaction. Length() 9879567 1. 37% 15. 70% G 4 Navigator: : Compute. Step() 9477618 1. 32% 17. 02% G 4 Stepping. Manager: : Set. Initial. Step() 9415974 1. 31% 18. 33% G 4 VEnergy. Loss. Process: : Along. Step. Do. It() 9217994 1. 28% 19. 61% G 4 Stepping. Manager: : Stepping() 8914560 1. 24% 20. 85% G 4 Klein. Nishina. Compton: : Sample. Secondaries() 8807404 1. 22% 22. 07% G 4 Navigator: : Locate. Global. Point. And. Update. Touchable. Handle() 7709725 1. 07% 23. 14% G 4 Stepping. Manager: : Invoke. PSDIP() 7634047 1. 06% 24. 20% CLHEP: : Hep. Rotation: : rotate. Axes() 7459101 1. 04% 25. 24% G 4 Quasmon: : Calculate. Hadronization. Probabilities() 7408432 1. 03% 26. 27% G 4 VEm. Process: : Post. Step. Do. It() 7311983 1. 02% 27. 28% __printf_fp 7178416 1. 00% 28. 28% G 4 Msc. Model: : Sample. Secondaries() 7137088 0. 99% 29. 27% G 4 QPDGCode: : Make. QCode() 6739533 0. 94% 30. 21% Stat. Accep. Test. Stepping. Action: : User. Stepping. Action() 6693000 0. 93% 31. 14% _int_malloc 6679076 0. 93% 32. 07% Stat. Accep. Test. Analysis: : classify. Particle() 6494007 0. 90% 32. 97% G 4 Navigation. Level. Rep: : G 4 Navigation. Level. Rep() 6434676 0. 89% 33. 86% G 4 Stepping. Manager: : Invoke. Along. Step. Do. It. Procs() 6393189 0. 89% 34. 75% G 4 Hadron. Cross. Sections: : Calc. Scattering. Cross. Sections() 6052991 0. 84% 35. 59% G 4 VEnergy. Loss. Process: : Post. Step. Do. It() 5487814 0. 76% 36. 36% G 4 Stepping. Manager: : Define. Physical. Step. Length() 5473295 0. 76% 37. 12% G 4 Electro. Nuclear. Cross. Section: : Get. Cross. Section() 5405793 0. 75% 37. 87% G 4 Msc. Model: : Geom. Path. Length() 5393849 0. 75% 38. 62% G 4 Transportation: : Post. Step. Do. It() 5114942 0. 71% 39. 33% G 4 QPDGCode: : Get. Nucl. Mass() Ryszard Jurga, CERN openlab 5044574 5024798 4880028 4662265 4508388 4409611 4403274 4088362 4081713 3890535 3837255 3809001 3775260 3681883 3631515 3587519 3550782 3538891 3398061 3394506 3346945 3303436 3280931 3275052 3238652 3222026 3190136 3181739 2939523 2936645 2906598 2796143 0. 70% 0. 68% 0. 65% 0. 63% 0. 61% 0. 57% 0. 54% 0. 53% 0. 52% 0. 51% 0. 50% 0. 49% 0. 47% 0. 46% 0. 45% 0. 44% 0. 41% 0. 40% 0. 39% 40. 03% 40. 73% 41. 41% 42. 05% 42. 68% 43. 29% 43. 90% 44. 47% 45. 04% 45. 58% 46. 11% 46. 64% 47. 17% 47. 68% 48. 18% 48. 68% 49. 17% 49. 67% 50. 14% 50. 61% 51. 08% 51. 53% 51. 99% 52. 45% 52. 90% 53. 34% 53. 79% 54. 23% 54. 64% 55. 04% 55. 45% 55. 84% Geant 4 Workshop, Oct. 2006 G 4 Hadronic. Process: : Fill. Total. Result() Stat. Accep. Test. Analysis: : info. Track() G 4 Replica. Navigation: : Back. Locate() G 4 QPDGCode: : Get. Quark. Content() G 4 QContent: : Get. SPDGCode() G 4 Voxel. Navigation: : Level. Locate() G 4 VMultiple. Scattering: : Get. Continuous. Step. Limit() G 4 Normal. Navigation: : Compute. Step() G 4 Transportation: : Along. Step. Do. It() G 4 QHadron: : Define. QC() CLHEP: : Ranlux. Engine: : flat() vfprintf G 4 Navigator: : Compute. Safety() G 4 Step. Point: : operator=() G 4 Sandia. Table: : Get. Sandia. Cof. Per. Atom() G 4 PEEffect. Model: : Sample. Secondaries() G 4 Tracking. Manager: : Process. One. Track() G 4 Multiple. Scattering: : True. Path. Length. Limit() G 4 Hadronic. Process: : General. Post. Step. Do. It() G 4 VEm. Process: : Get. Mean. Free. Path() G 4 QContent: : operator-=() G 4 Tubs: : Distance. To. Out() G 4 Bethe. Heitler. Model: : Sample. Secondaries() G 4 Photo. Nuclear. Cross. Section: : Get. Cross. Section() G 4 Tubs: : Inside() G 4 Particle. Change: : Check. It() G 4 QNucleus: : G 4 QNucleus() G 4 Msc. Model: : Sample. Displacement() G 4 LElastic: : Apply. Yourself() G 4 Particle. Change. For. Transport: : Update. Step. For. Along. Step() G 4 Physics. Vector: : Get. Value() G 4 Ion. Table: : Is. Light. Ion() 26

Further possible directions § icc test 40 Test. Em 3 -O 2 -prof-gen -ipo -prof-use Calorimeter … -O 3 § GNU test 40 Test. Em 3 -O 2 -fprofile-generate -fprofile-use Calorimeter … -O 3 Ryszard Jurga, CERN openlab § Geant 4 Workshop, Oct. 2006 already investigated 27

gcc 3. 2. 3 vs. icc 9. 0 § Ryszard Jurga, CERN openlab scaled in time, smaller is better Geant 4 Workshop, Oct. 2006 28

gcc comparison § Ryszard Jurga, CERN openlab scaled in time, smaller is better Geant 4 Workshop, Oct. 2006 29

G 4 shared libs vs. static libs § scaled in time, smaller is better Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 30

Future plans § Tool improve resolution of function names from shared libraries § number of function calls § § Geant 4 libs § different compilers • gcc, icc, open 64 § different compiler options § 32 bit and 64 bit mode Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 31

Questions and answers Q&A? Ryszard Jurga, CERN openlab Geant 4 Workshop, Oct. 2006 32