Proving cache coherence for the Alpha 21264 EV

Proving cache coherence for the Alpha 21264 (EV 6) processor Paul Harter, Leslie Lamport, Mark Tuttle, Yuan Yu Compaq Computer Corporation September 1999 Compaq Computer Corporation 1

Cache coherence protocols processor cache processor memory x=2 processor cache x=1 cache x=2 processor Alpha memory model defines ordering of reads and writes to x. Cache coherence protocol enforces the Alpha memory model. Goal: prove the cache coherence protocol is correct. September 1999 Compaq Computer Corporation 2

Proving cache coherence in “three easy steps”+“two-man years” Model Alpha memory model. (200 lines) Prove implementation (550 lines, 2 months, informal) Model abstract protocol. (500 lines) Prove implementation (5500 lines, 4+ months, incomplete) Model complete protocol. (2000 lines, 3 months) September 1999 Compaq Computer Corporation 3

Step 1: Alpha memory model We specify the Alpha memory model: – The official specification is an informal description of the allowed sequences of reads and writes. – We need a precise, state-based specification. We specify a simplified version of the model: – Operations read and write entire cache lines. – Operations accessing a cache line have a common point of synchronization. September 1999 Compaq Computer Corporation 4

Key definition: read/write ordering Before order for an execution orders reads/writes and determines what values are returned by reads. Good. Execution. Order defines good Before orders, namely the orders allowed by the memory model. September 1999 Compaq Computer Corporation 5

State machine actions Receive. Request(proc, req) Receive a request Choose. New. Data(proc, idx) Choose the return value for a request Respond(proc, idx) Return the value to a request Extend. Before Expand the Before relation Actions preserve Good. Execution. Order. September 1999 Compaq Computer Corporation 6

Good. Execution. Order This is the hard part --- but look how short it is! Good. Execution. Order == LET [some definitions deleted] IN / (*******************************) (* Before is a partial order. *) (*******************************) / Before subseteq Req. Id X Req. Id / A r 1, r 2 in Req. Id : Is. Before(r 1, r 2) => ~Is. Before(r 2, r 1) / A r 1, r 2, r 3 in Req. Id : Is. Before(r 1, r 2) / Is. Before(r 2, r 3) => Is. Before(r 1, r 3) / (*******************************) (* Source. Order implies the Before order. *) (*******************************) A r 1, r 2 in Req. Id : Source. Order(r 1, r 2) => Is. Before(r 1, r 2) / (*******************************) (* Request. Order implies the Before order. *) (*******************************) A r 1, r 2 in Req. Id : Request. Order(r 1, r 2) => Is. Before(r 1, r 2) September 1999 Compaq Computer Corporation 7

/ (****************************) (* Writes and successful SCs to the same location that *) (* have issued a response are totally ordered. *) (****************************) A r 1, r 2 in Req. Id : / Req. Id. Q[r 1]. req. type in {"Wr", "SC"} / Req. Id. Q[r 1]. req. new. Data # "Failed" / Req. Id. Q[r 1]. req. responded / Req. Id. Q[r 2]. req. type in {"Wr", "SC"} / Req. Id. Q[r 2]. req. new. Data # "Failed" / Req. Id. Q[r 2]. req. responded / Req. Id. Q[r 1]. req. adr = Req. Id. Q[r 2]. req. adr => Is. Before(r 1, r 2) / Is. Before(r 2, r 1) September 1999 Compaq Computer Corporation 8

/ (**********************************) (* LL/SC Axiom: For each successful SC, there is a matching LL and *) (* there is no write to the same address from a different *) (* processor between the LL and SC in the Before order. *) (**********************************) A r 2 in Req. Id : / Req. Id. Q[r 2]. req. type = "SC" / Req. Id. Q[r 2]. new. Data notin {Failed, Not. Chosen} => E r 1 in Req. Id : / LLSCPair(r 1, r 2) / A r in Req. Id : / / Req. Id. Q[r]. req. type = "Wr" / / Req. Id. Q[r]. req. type = "SC" / Req. Id. Q[r]. new. Data notin {Not. Chosen, Failed} / r[1] # r 2[1] / Req. Id. Q[r 2]. req. adr = Req. Id. Q[r]. req. adr => ~Is. Before(r 1, r) / ~Is. Before(r, r 2) September 1999 Compaq Computer Corporation 9

/ (*******************************) (* Value Axiom: A reads from the preceding write in the *) (* Before order. *) (*******************************) A r 1, r 2 in Req. Id : / Req. Id. Q[r 2]. source # No. Source / Req. Id. Q[r 1]. req. type = "Wr" / Req. Id. Q[r 1]. req. adr = Req. Id. Q[r 2]. req. adr => IF Req. Id. Q[r 2]. source = From. Init. Mem THEN ~Is. Before(r 1, r 2) ELSE / ~Is. Before(Req. Id. Q[r 2]. source, r 1) / ~Is. Before(r 1, r 2) September 1999 Compaq Computer Corporation 10

Step 2: Model abstract protocol Like most systems, the actual protocol is an – abstract protocol together with lots of – implementation details Unlike most systems, – abstract protocol’s correctness was far from obvious – we discovered a behavior not allowed by the model – this turned out to be an error in the memory model September 1999 Compaq Computer Corporation 11

The high-level proof Define protocol’s Before ordering: fairly easy. Prove it satisfies Good. Execution. Order: hard part was proving that the ordering is acyclic. Engineers had a behavioral intuition. Writing invariance proof was extremely hard: – 35 -line invariant, based on 300 lines of definitions – 550 -line proof, cases nested 10 levels deep September 1999 Compaq Computer Corporation 12

Step 3: Model complete protocol Obstacle 1: find a single, complete description – English documents: 20 documents, 4 -inch stack – Lisp simulator: crucial to understanding some details No description is – complete, precise, or – mathematically-tractable We wrote a relatively elegant, compact description September 1999 Compaq Computer Corporation 13

Step 3: Model complete protocol Obstacle 2: algorithm complexity – 60 different kinds of messages Quarks were the solution: – 15 units of functionality – each message modeled as a set of quarks – resolved message overloading, simplified protocol Protocol took 9 man-months, 1900 lines of TLA+ September 1999 Compaq Computer Corporation 14

The low-level proof Complete proof impossible due to time and labor Informal invariant was 1000 lines long We focus on the two most difficult conjuncts (each 150 lines) cache September 1999 messages point of data structure synch. Compaq Computer Corporation 15

The low-level proof Proof took 7 man-months – one conjunct: 2000 lines, cases 13 levels deep – second conjunct: potentially twice as long, stopped at a point of diminishing returns Found one actual error: – demonstration requires use of 4 processors, 2 memory locations, and 15 messages – state space is too big for model checkers to find it – error is too obscure for testing to find it September 1999 Compaq Computer Corporation 16

Lessons learned • Engineers can read TLA+ after an hour, write TLA+ after several hours • Engineers valued the work: the resulting confidence in the protocol was “invaluable” • Specification should be part of design process: – removes ambiguity, uncovers corner cases – describes entire system at single level of abstraction – allows use of tools like TLC early in design stage September 1999 Compaq Computer Corporation 17

Future work • Engineers – see the potential of formal methods – open to including formal methods in design phase • We want to facilitate adoption by engineering • Most likely future project: analyze proposals made to standards committees – PCI-X, … September 1999 Compaq Computer Corporation 18