Warp Pool Sharing Requests with InterWarp Coalescing for

Introduction • GPUs have high peak performance • For many benchmarks, memory throughput limits

GPU Architecture • 32 threads grouped into SIMD warps warp add r 1, r

GPU Memory System Warp Scheduler Load/Store Unit Intra-Warp Coalescer L 1 Load Group by

Problem: Divergence Warp Scheduler Load/Store Unit Intra-Warp Coalescer L 1 Load Group by cache

Problem: Bottleneck at L 1 Warp Scheduler Loads Warp 0 Warp 1 Warp 2

Hazards in Benchmarks 30 Cache lines per load/store Waiting loads/stores 25 20 15 10

Inter-Warp Spatial Locality • Spatial locality not just within a warp 0 warp 1

Inter-Warp Window Warp Scheduler Intra-Warp Coalescer 32 addresses Warp Scheduler 32 addresses Intra-Warp Coalescer

Design Overview Warp Scheduler Intra-Warp Coalescers Warp Scheduler Inter-Warp Coalescer Inter-Warp Queues L 1

Intra-Warp Coalescers Address Generation Warp Scheduler load . . . load Intra-Warp Coalescer to

Inter-Warp Coalescer intra-warp coalescers W 0 Cache line address W 0 . . .

Inter-Warp Coalescer intra-warp coalescers W 0 W 0 Cache line address warp ID .

Inter-Warp Coalescer intra-warp coalescers W 1 Cache line address W 1 warp ID .

Inter-Warp Coalescer intra-warp coalescers Cache line address warp ID . . . thread mapping

Selection Logic • Select a cache line from the inter-warp queues to send to

Methodology • Implemented in GPGPU-sim 3. 2. 2 • • GTX 480 baseline 32

Results: Speedup 5. 16 2. 35 3. 17 Speedup (x) 2 1. 38 x

Results: L 1 Throughput 1 0. 5 G AV lb m GE Cache-Limited SU

Results: L 1 Misses 100% % Baseline MPKI 75% 50% 25% Warp. Pool •

Conclusion • Many kernels limited by memory throughput • Key insight: use inter-warp spatial

Slides: 24

Download presentation

Warp. Pool: Sharing Requests with Inter-Warp Coalescing for Throughput Processors John Kloosterman, Jonathan Beaumont, Mick Wollman, Ankit Sethia, Ron Dreslinski, Trevor Mudge, Scott Mahlke Computer Engineering Laboratory University of Michigan

Introduction • GPUs have high peak performance • For many benchmarks, memory throughput limits performance 50% % Benchmarks 40% 30% 20% 10% 0% < 12% 12 -33% 33 -66% % cycles stalled 66%+ 2

GPU Architecture • 32 threads grouped into SIMD warps warp add r 1, r 2, r 3 thread load [r 1], r 2 • Warp scheduler sends ready warps to FUs warp 0 1 2 47. . . warp scheduler ALUs Load/Store Unit 3

GPU Memory System Warp Scheduler Load/Store Unit Intra-Warp Coalescer L 1 Load Group by cache line Cache Lines MSHR to L 2, DRAM 4

Problem: Divergence Warp Scheduler Load/Store Unit Intra-Warp Coalescer L 1 Load Group by cache line Cache Lines MSHR to L 2, DRAM … 5

Problem: Bottleneck at L 1 Warp Scheduler Loads Warp 0 Warp 1 Warp 2 Warp 3 Warp 4 Warp 5 Load/Store Unit Intra-Warp Coalescer Group by cache line Warp 1 0 Warp 5 Warp 4 L 1 Warp 3 Warp 2 MSHR to L 2, DRAM 6

Hazards in Benchmarks 30 Cache lines per load/store Waiting loads/stores 25 20 15 10 5 Memory Divergent Bandwidth-Limited G AV lb m SU M M V _2 GE BI CG M VT _1 _4 RR CO km ea ns _2 _1 AX AT _3 RR CO _1 M 2 M M M GE _1 M 3 M sc m v sp 3 g_ m ri- RK SY _1 pf SY R 2 K 0 Cache-Limited 7

Inter-Warp Spatial Locality • Spatial locality not just within a warp 0 warp 1 divergent inside a warp 2 warp 3 warp 4 8

Inter-Warp Spatial Locality • Spatial locality not just within a warp 0 warp 1 warp 2 warp 3 warp 4 9

Inter-Warp Spatial Locality • Spatial locality not just within a warp 0 warp 1 warp 2 warp 3 warp 4 • Key insight: use this locality to address throughput bottlenecks 10

Inter-Warp Window Warp Scheduler Intra-Warp Coalescer 32 addresses Warp Scheduler 32 addresses Intra-Warp Coalescer L 1 1 cache line from one warp Inter-Warp Coalescer 1 many cache line from cache lines one fromwarp many warps L 1 1 cache line from many warps 11

Design Overview Warp Scheduler Intra-Warp Coalescers Warp Scheduler Inter-Warp Coalescer Inter-Warp Queues L 1 Selection Logic L 1 12

Intra-Warp Coalescers Address Generation Warp Scheduler load . . . load Intra-Warp Coalescer to inter-warp coalescer Queue memory instructions • Queue load instructions before address generation • Intra-warp coalescers same as baseline • 1 request for 1 cache line exits per cycle 13

Inter-Warp Coalescer intra-warp coalescers W 0 Cache line address W 0 . . . Cache line address warp ID thread mapping . . . sort by address • Many coalescing queues, small # tags each • Requests mapped to coalescing queues by address • Insertion: tag lookup, max 1 per cycle per queue 14

Inter-Warp Coalescer intra-warp coalescers W 0 W 0 Cache line address warp ID . . . Cache line address thread mapping warp ID thread mapping . . 0 . . . sort by address • Many coalescing queues, small # tags each • Requests mapped to coalescing queues by address • Insertion: tag lookup, max 1 per cycle per queue 15

Inter-Warp Coalescer intra-warp coalescers W 1 Cache line address W 1 warp ID . . . thread mapping warp ID thread mapping 0 0 . . . Cache line address . . sort by address • Many coalescing queues, small # tags each • Requests mapped to coalescing queues by address • Insertion: tag lookup, max 1 per cycle per queue 16

Inter-Warp Coalescer intra-warp coalescers Cache line address warp ID . . . thread mapping Cache line address warp ID 0 0 1 . . . thread mapping . . . sort by address • Many coalescing queues, small # tags each • Requests mapped to coalescing queues by address • Insertion: tag lookup, max 1 per cycle per queue 17

Selection Logic • Select a cache line from the inter-warp queues to send to L 1 Cache . . . Selection Logic • 2 strategies: • Default: pick oldest request • Cache-sensitive: prioritize one warp • Switch based on miss rate over quantum 18

Methodology • Implemented in GPGPU-sim 3. 2. 2 • • GTX 480 baseline 32 MSHRS 32 k. B cache GTO scheduler • Verilog implementation for power and area • Benchmark criteria • Parboil, Poly. Bench, Rodinia benchmark suites • Memory throughput limited: waiting memory requests for more than 90% of execution time • Warp. Pool configuration • • 2 intra-warp coalescers 32 inter-warp queues 100, 000 cycle quantum for request selector Up to 4 inter-warp coalesces per L 1 access 19

Results: Speedup 5. 16 2. 35 3. 17 Speedup (x) 2 1. 38 x 1. 5 1 0. 5 8 -way banked cache [1] MRPB: Memory request prioritization for massively parallel processors: HPCA 2014 N OM EA lb m GE SU M M V _2 Cache-Limited GE BI CG M VT _1 _4 RR CO ns ea AX AT MRPB _2 _1 _3 RR M 2 M CO _1 M M Bandwidth-Limited km Memory Divergent GE _1 M 3 M sc m v sp 3 g_ m ri- RK SY _1 pf SY R 2 K 0 Warp. Pool 20

Results: L 1 Throughput 1 0. 5 G AV lb m GE Cache-Limited SU M M V _2 BI CG M VT _1 _4 RR ns ea CO _2 _1 km AT Bandwidth-Limited 8 -way banked cache AX _3 RR CO _1 M 2 M M M GE _1 M 3 M Memory Divergent sc m v sp 3 g_ m ri- RK SY _1 pf R 2 K 0 SY Requests Serviced per L 1 access 1. 5 Warp. Pool • Banked cache uses divergence, not locality • Warp. Pool merges even when not divergent • No speedup for banked cache: 1 miss/cycle 21

Results: L 1 Misses 100% % Baseline MPKI 75% 50% 25% Warp. Pool • MRPB has larger queues • Oldest policy sometimes preserves cross-warp temporal locality [1] MRPB: Memory request prioritization for massively parallel processors: HPCA 2014 N GE OM EA lb m SU M M V _2 Cache-Limited GE BI CG M VT _1 _4 RR CO _2 ns ea km AX AT Bandwidth-Limited MRPB _1 _3 CO RR _1 M 2 M M M GE _1 M 3 M Memory Divergent sc m v sp 3 g_ rim RK SY _1 pf SY R 2 K 0% 22

Conclusion • Many kernels limited by memory throughput • Key insight: use inter-warp spatial locality to merge requests • Warp. Pool improves performance by 1. 38 x: • Merging requests: increase L 1 throughput by 8% • Prioritizing requests: decrease L 1 misses by 23% 23