Finding Oil with Cells Seismic Imaging Using a

Finding Oil with Cells: Seismic Imaging Using a Cluster of Cell Processors Michael Perrone IBM Master Inventor Mgr, Multicore Computing, IBM Research © 2009 IBM Corporation

IBM Research Outline § Problem description § Algorithmic solution § Cell implementation issues § Conclusions 2 © 2009 IBM Corporation

IBM Research Challenges 150 M$ § § 3 60 -70% “Easy” oil is gone Focus on imaging Mathematics is known Computer power isn’t affordable yet © 2009 IBM Corporation

IBM Research Salt Domes § Gulf of Mexico § Deep sea § ~50 B barrels § Difficult to image! § http: //www-sst. unil. ch/research/seismic/G_Mexico. htm 4 © 2009 IBM Corporation

IBM Research Sample Image 5 © 2009 IBM Corporation

IBM Research Snell’s Law a a Reflected wave Incident wave V 1 V 2 Transmitted wave V 3 Boundaries 6 © 2009 IBM Corporation

IBM Research Seismic Data Collection Air Gun Surface Receiver Buried reflectors 7 © 2009 IBM Corporation

IBM Research Seismic Data Collection Air Gun Surface Receiver Buried reflectors 8 © 2009 IBM Corporation

IBM Research Seismic Data Collection Air Gun Surface Receiver Buried reflectors Internal Reflections 9 © 2009 IBM Corporation

IBM Research Ocean Seismic Survey Streamers § 4, 8, 16 or more streamers § Receiver every ~25 meters § Dragged across survey area Ship 1 km 5 km 1 Shot 10 © 2009 IBM Corporation

IBM Research Ocean Seismic Survey § § Multiple passes Drift Noise Equipment failure § 10 -20 TB per survey § >100 K Shots recorded 11 © 2009 IBM Corporation

IBM Research Outline § Problem description § Algorithmic solution § Cell implementation issues § Conclusions 12 © 2009 IBM Corporation

IBM Research RTM - Reverse Time Migration (Biondi & Shan, 2002) § Use 3 D wave equation to model sound in Earth [ ∂2 t - v 2(x, y, z) (∂2 x + ∂2 y + ∂2 z) ] P(x, y, z, t) = S(x, y, z, t) § Forward Propagation: F(x, y, z, t) Reverse Propagation: R(x, y, z, t) § Image generation I(x, y, z) = 13 ∑t F(x, y, z, t) R(x, y, z, t) © 2009 IBM Corporation

IBM Research Outline § Problem description § Algorithmic solution § Cell implementation issues § Conclusions 14 © 2009 IBM Corporation

IBM Research Implementing the Wave Equation § Finite difference in time: µ ∂ 2 t P(x, y, z, t) ≈ P(x, y, z, t+1) - 2 P(x, y, z, t) + P(x, y, z, t-1) § Finite difference in space: ∂2 x P(x, y, z, t) ≈ ∑ngx(n)P(x+n, y, z, t) ∂2 y P(x, y, z, t) ≈ ∑ngy(n)P(x, y+n, z, t) ∂2 z P(x, y, z, t) ≈ ∑ngz(n)P(x, y, z+n, t) § Absorbing boundary conditions 15 © 2009 IBM Corporation

IBM Research Data Flow § Hide disk access behind computes § Re-use data - Tiling § Main memory (each iteration) ► Load – V(x, y, z) – P(x, y, z, t-1) ► Store P(x, y, z, t+1) § Disk (every Nth iterations) ► Store P(x, y, z, t) - or ► Load P(x, y, z, t) § Multibuffer DMAs § Load Balancing 16 © 2009 IBM Corporation

IBM Research Image RTM Algorithm (for each shot) § Load data ► Velocity model v(x, y, z) t=N F(N) R(N) I(N) ► Calculate F(x, y, z, t) t=2 N F(2 N) R(2 N) I(2 N) ► Every N timesteps – Compress F(x, y, x, t) – Write F(x, y, x, t) to disk/memory t=3 N F(3 N) R(3 N) I(3 N) . . . t=k. N F(k. N) ► Source & Receiver data § Forward propagation § Backward propagation ► Calculate R(x, y, z, t) ► Every N timesteps – Read F(x, y, x, t) from disk/memory – Decompress F(x, y, x, t) – Calculate partial sum of I(x, y, z) § Merge I(x, y, z) with global image 17 . . . R(k. N) © 2009 IBM Corporation I(k. N)

IBM Research Data Partitioning § Simple split on XZ-plane § Boundary communication with DMAs or MPI ► Send boundary ► Compute bulk Hides Latency ► Receive boundary ► Compute boundary X Y Z 1 18 2 © 2009 IBM Corporation

IBM Research Load Balancing: Source Data “Spray” Full Velocity Model X Shot Y Spray Velocity Subset § Problem: BW bottleneck 19 § Solution: Improve data access ► “Spray” shot data onto grid ► Sort data points ► Irregular data access ► Calculate once per shot ► Z direction stride 1 ► Adjust partitioning to balance © 2009 IBM Corporation

IBM Research Data Compression “Global” “Local” Z § Compression ► Lossy Max/Min scaling ► Signal decays with depth ► 4 x ► Local version gives better signal resolution ► Reduced BW requirement 20 § Quality © 2009 IBM Corporation

IBM Research RTM Parameters § Survey size ► Frequency: ~5 shots / minute ► Shots: >100 K per survey ► Survey size: 10 -20 TB ► Wall time: ~1 month § Velocity model ► Multiple square miles: 2 x 2, 5 x 5, 10 x 10, etc. ► 1 -10 GB § Migration iterations ► Thousands ► Depends on depth required 21 © 2009 IBM Corporation

IBM Research “Master/Slave” Cell Cluster Building Blocks SRW 248 Compute racks (72 QS 22’s) plus storage rack 22 © 2009 IBM Corporation

IBM Research Outline § Problem description § Algorithmic solution § Cell implementation issues § Conclusions 23 © 2009 IBM Corporation

IBM Research 150 M$ 60 -70% Conclusions § Built 297 -node Cell cluster § Demonstrated ► Effectiveness of Master/Slave model ► Data flow issues can be managed ► Significant speed-up over Intel (4 -10 x depending on data) § Currently generating real seismic images! Will be used to determine drilling sites…. [ ∂2 t - v 2(x, y, z) (∂2 x + ∂2 y + ∂2 z) ] P(x, y, z, t) = S(x, y, z, t) 24 © 2009 IBM Corporation

IBM Research BACKUP SLIDES 25 © 2009 IBM Corporation

IBM Research Abstract § Modern deep sea oil exploration is a very expensive proposition. A single well can cost about 150 M$ and the probability of drilling a "dry" hole is about 60 -70 percent! To help reduce cost, oil companies have turned to increasingly complex computational imaging techniques to improve the quality of imaging. And in order to reduce the "time to oil", images must be generated as quickly as possible; so these algorithms are run on high-performance computing clusters. This presentation will discuss one such imaging application implemented on a 296 node, heterogeneous cluster composed primarily of Cell processors. 26 © 2009 IBM Corporation