Interplay between Hardware Prefetcher and Page Eviction Policy

Interplay between Hardware Prefetcher and Page Eviction Policy in CPU-GPU Unified Virtual Memory ISCA 2019 Debashis Ganguly Ziyu Zhang, Jun Yang, Rami Melhem

Why do we need Hardware Prefetchers? Kernel Execution Far fault Data Migration 2

Why do we need Hardware Prefetchers? Kernel Execution Far fault Data Migration Stream 0 Stream 1 User Directed Prefetch 2

Why do we need Hardware Prefetchers? Kernel Execution Far fault Data Migration Stream 0 Stream 1 User Directed Prefetch • What and when to prefetch? • How do I synchronize between streams? 2

Why do we need Hardware Prefetchers? Kernel Execution Far fault Data Migration Stream 0 Stream 1 User Directed Prefetch • What and when to prefetch? • How do I synchronize between streams? Hardware Prefetch 2

Why do we need Hardware Prefetchers? Kernel Execution Far fault Data Migration Stream 0 Stream 1 User Directed Prefetch • What and when to prefetch? • How do I synchronize between streams? • Takes away the programming effort • Follows spatio-temporal locality of past accesses • Overlap kernel execution and data migration Hardware Prefetch 2

Different Hardware Prefetchers Random Prefetcher (Rp) 2 MB 2 MB Randomly prefetch a 4 KB page local to the 2 MB large page to which the current faulty page belongs 3

Different Hardware Prefetchers Random Prefetcher (Rp) 2 MB 2 MB Randomly prefetch a 4 KB page local to the 2 MB large page to which the current faulty page belongs Sequential-local 64 KB Prefetcher (SLp) [Variation of Sequential and Locality-aware] 2 MB 64 K B Prefetch 64 KB basic block corresponding to which the current faulty page belongs 3

Tree-based Neighborhood Prefetcher (TBNp) Invalid Page Access Far fault Prefetch 0% 0% 0% 0% 64 K 64 K 4

Tree-based Neighborhood Prefetcher (TBNp) Invalid Page Access Far fault Prefetch 12. 5% 0% 25% 0% 64 K 100% 0% 4 K 60 K 0% 0% 0% 50% 0% 64 K 64 K 64 K 1 1 4

Tree-based Neighborhood Prefetcher (TBNp) Invalid Page Access Far fault Prefetch 25% 50% 0% 64 K 100% 0% 4 K 60 K 100% 0% 0% 64 K 4 K 1 1 0% 0% 60 K 0% 0% 64 K 64 K 2 2 4

Tree-based Neighborhood Prefetcher (TBNp) Invalid Page Access Far fault Prefetch 37. 5% 0% 75% 100% 50% 100% 0% 4 K 60 K 100% 0% 4 K 100% 0% 0% 64 K 4 K 1 3 3 60 K 1 0% 0% 60 K 0% 0% 64 K 64 K 2 2 4

Tree-based Neighborhood Prefetcher (TBNp) Invalid Page Access Far fault Prefetch 50% 0% 100% 0% 4 K 60 K 100% 0% 4 K 3 3 1 100% 0% 60 K 64 K 1 4 0% 0% 100% 0% 4 K 60 K 0% 0% 64 K 64 K 2 2 4

Tree-based Neighborhood Prefetcher (TBNp) Invalid Page Access Far fault Prefetch 62. 5% 25% 100% 0% 4 K 60 K 100% 0% 4 K 3 3 1 100% 0% 60 K 64 K 1 4 0% 50% 100% 0% 4 K 60 K 100% 0% 4 K 2 2 60 K 0% 0% 0% 64 K 64 K 5 4 4

Tree-based Neighborhood Prefetcher (TBNp) Invalid Page Access Far fault Prefetch 100% 100% 0% 4 K 60 K 100% 0% 4 K 3 3 1 100% 0% 60 K 64 K 1 4 100% 0% 4 K 60 K 100% 0% 4 K 2 2 0% 100% 60 K 64 K 64 K 5 5 4 4

When working set fits in device memory TBNp has 1 -2 order of magnitude performance improvement over no prefetching • Larger the transfer size, higher the bandwidth • Reduced number of far-faults 5

What happens under device memory oversubscription? • Disable hardware prefetchers • To avoid displacement of heavily referenced pages • Pre-eviction to maintain free-page buffer • To avoid write-back latency Early disabling of prefetcher by pre-eviction ~100 x performance degradation with just 110% oversubscription 6

Interplay between Prefetcher and Naïve Eviction Policies LRU 4 KB 2 MB 64 KB 64 KB 7

Interplay between Prefetcher and Naïve Eviction Policies LRU 4 KB 2 MB 64 KB 64 KB • No contiguous free space to prefetch • Renders prefetcher ineffective 7

Interplay between Prefetcher and Naïve Eviction Policies LRU 4 KB 2 MB 64 KB LRU 2 MB 64 KB 64 KB 2 MB 64 KB • No contiguous free space to prefetch • Renders prefetcher ineffective 7

Interplay between Prefetcher and Naïve Eviction Policies LRU 4 KB 2 MB 64 KB LRU 2 MB 2 MB 64 KB 64 KB 2 MB 64 KB • No contiguous free space to prefetch • Renders prefetcher ineffective 7

Interplay between Prefetcher and Naïve Eviction Policies LRU 4 KB 2 MB 64 KB LRU 2 MB 2 MB 64 KB • No contiguous free space to prefetch • Renders prefetcher ineffective 64 KB 2 MB 64 KB • Displace heavily referenced pages • Causes large thrashing 7

Prefetcher Inspired Eviction Policies Random Eviction (Re) 2 MB 2 MB Randomly evict a 4 KB page from the entire virtual address space 8

Prefetcher Inspired Eviction Policies Random Eviction (Re) 2 MB 2 MB Randomly evict a 4 KB page from the entire virtual address space Sequential-local 64 KB Pre-eviction (SLe) 2 MB 64 K B Pre-evict 64 KB basic block corresponding to the 4 KB LRU candidate 8

Tree-based Neighborhood Pre-eviction (TBNe) Valid LRU Candidate LRU Eviction Pre-eviction 100% 100% 100% 100% 64 K 64 K 9

Tree-based Neighborhood Pre-eviction (TBNe) Valid LRU Candidate LRU Eviction Pre-eviction 87. 5% 100% 75% 100% 64 K 100% 0% 4 K 60 K 100% 50% 100% 100% 64 K 64 K 64 K 1 1 9

Tree-based Neighborhood Pre-eviction (TBNe) Valid LRU Candidate LRU Eviction Pre-eviction 75% 100% 50% 100% 64 K 100% 0% 4 K 60 K 0% 100% 64 K 4 K 1 1 100% 50% 60 K 100% 64 K 64 K 2 2 9

Tree-based Neighborhood Pre-eviction (TBNe) Valid LRU Candidate LRU Eviction Pre-eviction 62. 5% 75% 50% 100% 64 K 100% 0% 4 K 60 K 0% 100% 64 K 4 K 1 1 100% 50% 50% 60 K 0% 100% 4 K 2 2 60 K 100% 64 K 64 K 3 3 9

Tree-based Neighborhood Pre-eviction (TBNe) Valid LRU Candidate LRU Eviction Pre-eviction 50% 75% 25% 100% 0% 4 K 60 K 100% 0% 4 K 0% 100% 64 K 4 K 1 4 4 60 K 1 100% 50% 0% 60 K 0% 100% 4 K 2 2 60 K 100% 64 K 64 K 3 3 9

Tree-based Neighborhood Pre-eviction (TBNe) Valid LRU Candidate LRU Eviction Pre-eviction 37. 5% 75% 0% 100% 0% 4 K 60 K 100% 0% 4 K 4 4 1 0% 100% 0% 60 K 64 K 1 5 100% 50% 0% 0% 4 K 60 K 0% 100% 4 K 2 2 60 K 100% 64 K 64 K 3 3 9

Tree-based Neighborhood Pre-eviction (TBNe) Valid LRU Candidate LRU Eviction Pre-eviction 0% 0% 0% 100% 0% 4 K 60 K 100% 0% 4 K 4 4 0% 0% 0% 100% 0% 60 K 64 K 1 5 4 K 60 K 0% 100% 4 K 2 2 0% 0% 100% 60 K 64 K 64 K 3 6 6 6 3 9

Combining Pre-evictions (4 KB Granularity) and Prefetchers Order of magnitude performance improvement by TBNp and TBNe combo • No additional co-ordination required • Respecting each other pays off 10

Combining Pre-evictions (2 MB Granularity) and Prefetchers Average 18. 5% performance improvement by TBNe • Dynamic eviction granularity • Reduced number of thrashing 11

Conclusion • Leverages the framework for hardware prefetcher • No additional implementation and performance overhead • Builds on generic concepts • Vendor agnostic • Opportunistically decide on dynamic eviction granularity • Navigates between two extremes: 4 KB and 2 MB • Overcomes limitations with static granularity • Micro-benchmarks, UVM benchmarks, and simulator • Public for future collaboration • https: //github. com/Debashis. Ganguly/gpgpu-sim_UVMSmart 12

Interplay between Hardware Prefetcher and Page Eviction Policy in CPU-GPU Unified Virtual Memory Debashis Ganguly Ph. D. Student debashis@cs. pitt. edu https: //people. cs. pitt. edu/~debashis/