Life Beyond Distributed Transactions An Apostates Opinion By

Life Beyond Distributed Transactions An Apostate’s Opinion By Pat Helland Amazon. Com Jan 8 th, 2007 Apostate: noun “One who renounces a previously held belief. ” Today’s Goal: Offer hopefully insightful opinions about scaleable apps. incite-ful 1

Assumptions (Don’t Gotta Prove These… Just Plain Believe Then) Grown-Ups Don’t Use Distributed Transactions • The apps using distributed transactions become too fragile… • Let’s just consider local transactions. Multiple Disjoint Scopes of Serializability Want Almost-Infinite Scaling • More of everything… Year by year, bigger and bigger • If it fits on your machines, multiply by 10, if that fits, multiply by 1000… • Strive to scale almost linearly (N log N for some big log). Want Scale-Agnostic Applications • Two layers to the application: scale-agnostic and scale-aware • Consider Scale-Agnostic API Application Scale Agnostic Code Scale Agnostic API Upper Layer Lower Layer Scale-Aware-Code 2

Uniquely Keyed Entities • Not All Data May Be in a Single Transaction – We Must Collect the Data into Pieces – We Must Annotate the Boundaries of the Data Guaranteed to Be Transactional • Must Remain Transactional Even If We Repartition! • An Entity: – A Collection of Data that Fits on a Single Machine – Identified by a Unique Key • Assume the Scale-Aware-Code Never Partitions an Entity – The Unique Key Defines the Data that Can’t Be Partitioned Key = “ABC” Entity Key = “WPB” Entity Key = “QLA” Entity Key = “UNB” Entity The Application’s Data Is Factored into Entities, Each of Which Has a Unique Key Each Entity Will Reside on a Single Machine (Ignoring Replication & H/A) 3

Transactions and Entities • A Transaction May Update a Single Entity – The Scale-Aware-Code (and API) Guarantee It • The Entity Is Never Partitioned • A Transaction Must Not Ever Update Two Entities – Even If the Two Live on One Machine Today – Tomorrow, They May Repartition to Different Machines… Entity “ABC” Transaction Entity “DEF” 4

Repartitioning and Entities • Entities Allow Scaling – Entities Remain Intact Even when Repartitioning • The Application Can Count on the Integral Nature of Each Entity – It Is OK to Know that the Entire Entity Is Local – It Is OK to Work on Anything in the Entity at Once Entity “ABC” Entity “FAW” “LMN” “EFG” “GHI” Entity “ABZ” Entity “FXQ” “JKL” Entity “GHI” Entity “MOE” Entity “NAO” Entity “DEF” Entity “LMN” “JAA” Entity “RST” “JKL” Entity “RAA” Entity “RST” Entity “XYZ” Entity “KZU” Entity “XYZ” “LMN” Entity “XYZ” No Promise that Two Different Entities Stay on the Same Machine!! 5

Thinking about Alternate Indices • Entities Must Have a Unique Key – Unless the You Begin with the Same Key, You Aren’t the Same • CANNOT Guarantee the Alternate Index Will Co-locate with the Entity’s Primary Key – By Definition, Alternate Indices Don’t Have the Same Key! – We Must Index Them with a Different Key… • Alternate Indices CANNOT Be Updated in the Same Transaction as the Primary Data – There Is No Way to Guarantee They Are on the Same Machine – They Must Be Updated in Different Transactions… A 2: klw A 2: ghu A 2: fgh A 2: def Entity Keys Indexed by 2 nd Alternate Key A 2: aba A 1: MNO A 1: JKL A 1: GHI A 1: DEF A 1: ABC Entity Keys Indexed by 1 st Alternate Key PK: 719 PK: 589 PK: 332 PK: 217 PK: 123 Entities Indexed by Unique Key 6

Entities Are Connected by Messaging • Entities Are Key-Named Boundaries for Transactional Work – Transactions Never Span Entities – The Scale-Aware-Code May Move Them to Repartition • The Only Way to Communicate across Entities Is with Messaging! – The Scale-Aware-Code Is Responsible for Finding the Correct Entity (by Key-Name) and for Routing the Message to It Entity-X Send To: Entity-Y Boundary of Transactions “Messaging” Is in Quotes… Work Is Invoked -- Potentially across Machines -- Definitely across Transactions! 7

Messages Connect Entities • Messages Are the Only Way into and out of Entities – They Are Produced by Transactions – They Are Consumed by Transactions – Transactions Are Local to the Entity From: Entity-A From: Entity-B From: Entity-C From: Entity-D Send To: Entity-A Entity-X Send To: Entity-B Send To: Entity-C 8

Entities Connected by Partnerships • Mostly, Messaging Occurs between Two Partner Entities – Usually, a Two-Way Exchange Moving Both Entities’ State – Each Keeps Data about How Far Its State Has Advanced… Entity-W Entity-X Entity-Y Entity-Z 9

Tracking a Partner with Activities • Activity Refers to the Knowledge about a Partner Entity – Descriptions of What Messages Have Been Received – Descriptions of What Obligations Exist to the Partner – The Foundation for Workflow to Replace Distributed Transactions Entity-W Activity-X Entity-X Activity-W Activity-Y Entity-Y Activity-X Activity-Z Activity-X Entity-Z • Two Basic Observations Wrapped Up in the “Activity” Concept – Work Across Entities Is Workflow Based on Two-Party Relationships – The Granularity of the Workflow Participant Is an Entity (Fine-Grained) 10

New Vocabulary for Discussing Scale Vocabulary and Assertions Almost-Infinite Scaling Scale-Agnostic App Entity Assertions about Large Scale Apps Activity An Environment Demanding Rapidly Increasing Data and Computation Over Time An Application that Does Not Need to Change to Support Almost-Infinite Scaling A Collection of Data Referenced by a Single Key; Transactional Scope of the Scale-Agnostic App Data Used Inside One Entity to Describe Its Workflow State with a Single Partner Entity Alternate Indices Aren’t Transactionally Consistent As Scale Increases, The Primary and Alternate Indices Cannot Be Guaranteed to Live Together Entities Cooperate Using Fine-Grained Two-Party Workflow No Dist-Txs Workflow; Workflow Participants Are Entities; Work Coordinated across Pairs 11