SHi P Signaturebased Hit Predictor for High Performance

SHi. P: Signature-based Hit Predictor for High Performance Caching *Carole-Jean *Margaret *Princeton Wu, #Aamer Jaleel, #, +William Hasenplaugh, Martonosi, #Simon Steely Jr. , University #Intel #, +Joel Corporation, VSSAD Emer #, +MIT IEEE/ACM International Symposium on Microarchitecture (MICRO’ 2011)

Motivation • Factors making caching important • Increasing ratio of CPU speed to memory speed • Multi-core poses challenges on better shared cache management • LRU has been the standard LLC replacement policy • However LRU has problems! 2

Problems with LRU Replacement Working set larger than the cache causes thrashing miss miss Wsize LLCsize References to non-temporal data (scans) discards frequently referenced working set hit Wsize hit scan LLCsize miss hit scan miss scans occur frequently in commercial workloads 3

Desired Behavior from Cache Replacement Wsize LLCsize hit miss hit miss Working set larger than the cache Preserve some of working set in the cache [ DIP (ISCA’ 07), DRRIP (ISCA’ 10) achieves this effect ] Recurring scans Preserve frequently referenced working set in the cache hit hit scan hit [ SRRIP (ISCA’ 10) achieves this effect ] 4

Dynamic Re-Reference Interval Prediction ( DRRIP ) (SRRIP) Scan-Resistant insertion 0 Immediate No Victim 1 Intermediate No Victim 2 far re-reference ( BRRIP ) Thrash-Resistant insertion No Victim 3 distant eviction re-reference [ Jaleel et al. , ISCA’ 10 ] 5

SRRIP Not Always Scan Resistant… • LONG scans in access pattern miss hit “short” scan hit “long” scan miss 6

SRRIP Not Always Scan Resistant… • LONG scans in access pattern miss • hit “short” scan hit “long” scan miss Active working-set MUST be RE-REFERENCED at least ONCE between scans miss scan miss 7

SRRIP Not Always Scan Resistant… • LONG scans in access pattern miss • hit “short” scan hit hit miss “long” scan Active working-set MUST be RE-REFERENCED at least ONCE between scans miss scan miss hit Can We Be More Intelligent in Dealing with Scans? 8

Closer Look at Scan Access Patterns scan Future Reference scan No Future References Assuming Perfect Knowledge of Re-Reference Pattern 9

Improving RRIP on Cache Insertions Improve Insertion 0 Immediate No Victim 1 Intermediate No Victim 2 far re-reference No Victim scan 3 distant eviction re-reference Need to Assign DIFFERENT Re-Reference Predictions on Cache Insertion 10

Focus of this Paper… • Goal: Learn re-reference interval of a cache line cache access PREDICTOR re-reference prediction How Best to Learn the Re-Reference Interval? 0: 1: 2: 3: immediate intermediate far distant 11

Learning Re-Reference Behavior scan REFERENCED BY SIMILAR SET OF PCs scan REFERENCE SAME MEMORY REGION Can We Learn Re-References By Correlating Accesses With Some Other Information? 12

Learning Re-Reference Behavior scan REFERENCED BY SIMILAR SET OF PCs scan REFERENCE SAME MEMORY REGION Can We Learn Re-References By Correlating Accesses With Some Other Information? 13

Using Signatures to Correlate Re-Reference scan • Different types of information: • Memory Region • Memory Instruction PC • Instruction Sequence • scan “signature“ Observation: LLC accesses by the same “signature” tend to have similar re-reference patterns OBSERVE, LEARN and PREDICT Re-Reference Pattern of a Signature

Observe Signature Re-Reference Behavior Load/Store Address • Observe re-reference pattern in the baseline cache • Cache Tag • Replacement State • Coherence State LLC 15

Observe Signature Re-Reference Behavior Load/Store • Was line re-referenced after cache insertion ( 1 -bit ) • “Signature” responsible for cache insertion ( 14 -bits ) Address • Hardware Required: Signature • Observe re-reference pattern in the baseline cache • reuse bit • signature_insert metadata LLC 16

Learn Signature Re-Reference Behavior • Learn signature re-reference behavior • Hardware Required: • Signature History Counter Table (SHCT) ( 16 K, 2 -bit counters ) • SHCT Training: counter = 0, signature NOT re-referenced counter != 0, signature re-referenced • If evicted line reused: SHCT [ signature_insert ] ++ SHCT • If evicted line NOT reused: SHCT [ signature_insert ] -Last Level Cache (LLC) 17

Signature-based Hit Predictor (SHi. P) • Predict re-reference interval of line using SHCT cache hit/miss signature SHi. P SHCT re-reference prediction 0: 1: 2: 3: immediate intermediate far distant 18

Signature-based Hit Predictor (SHi. P) • Predict re-reference interval using SHCT on CACHE MISS cache miss signature SHi. P Re-Reference Predictions On Miss if ( SHCT [ signature ] == 0 ) predict DISTANT (i. e. 3) else predict FAR (i. e. 2) re-reference prediction 0: 1: 2: 3: immediate intermediate far distant 19

Signature-based Hit Predictor (SHi. P) • Predict re-reference interval on CACHE HIT cache hit signature SHi. P Re-Reference Predictions On Hit Always predict IMMEDIATE (i. e. 0) re-reference prediction 0: 1: 2: 3: immediate intermediate far distant 20

SHCT SHi. P hit/miss data Access Type Address Signature SHi. P – High Level Architectural Overview SHCT Training signature_insert reuse_bit LLC hit/miss Re-Reference Prediction Last Level Cache (LLC) 21

SHCT SHi. P hit/miss data Address Signature Per-Line Overhead Can Be Reduced by using Set Sampling ( need only 32 - 64 sets ) Access Type SHi. P – High Level Architectural Overview SHCT Training signature_insert reuse_bit LLC hit/miss Re-Reference Prediction Last Level Cache (LLC) 22

~6 KB SHCT Training signature_insert reuse_bit LLC hit/miss SHi. P Re-Reference Prediction hit/miss data Address Signature Per-Line Overhead Can Be Reduced by using Set Sampling ( need only 32 - 64 sets ) Access Type SHi. P – High Level Architectural Overview E G N A H C O N Last Level Cache (LLC) 23

Performance Comparison of Replacement Policies Performance Relative to LRU 1. 15 16 -way 2 MB LLC Core i 7 Type Hierarchy SRRIP DRRIP SHi. P-PC 1. 10 1. 05 1. 00 Mm. /Games Server SPEC 2 K 6 All SHi. P Significantly Improves Performance Across All Workload Categories 24

Performance Relative to LRU Performance Comparison of Replacement Policies CRC Results Comparison 1. 15 SRRIP 1. 10 DRRIP Seg-LRU SDBP Averaged Across PC Games, Multimedia, Enterprise Server, SPEC CPU 2006 Workloads 1. 05 SHi. P-PC S H i P 1. 00 16 -way 1 MB Private Cache 16 -way 4 MB Shared Cache 65 Single-Threaded Workloads 165 4 -core Workloads SHi. P Has 2 X Performance Improvements of Prior State-of-the-Art Policies 25

Total Storage Overhead (16 -way Set Associative Cache) • LRU: • Pseudo-LRU • RRIP: • Seg-LRU: 4 -bits / cache block 1 -bit / cache block [ ISCA’ 10 ]2 -bits / cache block [ CRC’ 10 ] ~8 -bits / cache block • SDBP: [ MICRO’ 10 ] ~10 -bits / cache block • SHi. P: [ MICRO’ 11 ] ~5 -bits / cache block SHi. P Outperforms State-of-the-Art with HW Similar to LRU 26

Summary • Scan-resistance is an important problem in commercial workloads • State-of-the art policies do not fully address scan-resistance • Signatures help improve re-reference predictions to address scans • Need fine-grained re-reference predictions at insertion • Proposed a Simple and Practical Scan-Resistant Replacement • SHi. P significantly outperforms winner of CRC Championship • SHi. P requires less storage than CRC winner • HW overhead of SHi. P is comparable to LRU 27

Q&A 28

Q&A 29

Q&A 30

Re-Reference Interval Prediction ( RRIP ) CAN INSERTION BE Scan-Resistant MORE INTELLIGENT? insertion 0 Immediate No Victim 1 Intermediate No Victim 2 far re-reference No Victim 3 distant eviction re-reference 31

Using Signatures to Correlate Re-Reference Behavior SIGN ATURE a b a c scan d c scan Example Signatures Memory Region Program Counter Instruction Decode History Future Cache Hits No Future Cache Hits a c b d 32

LRU vs. Re-Reference Interval Prediction (RRIP) Physical Way # 0 1 2 3 4 5 6 7 Cache Tag s c b h f d g e 2 2 1 3 0 5 4 7 6 “LRU Chain” position LRU RRIP Outperforms LRU with Storage Less Than LRU Physical Way # 0 1 2 3 4 5 6 7 Cache Tag s c b h f d g e Re-Reference Prediction 0 0 2 1 0 2 2 3 3 RRIP 33

Signature-based Hit Predictor (SHi. P) LLC hit/miss data Access Type Learn Re-Reference Behavior: Address • Predict the re-reference behavior of a signature Signature • Goal: 34