Distributed Transactions Are Dead Long Live Distributed Transactions
Distributed Transactions Are Dead, Long Live Distributed Transactions! Support for transactions in Orleans 2. x Sergey Bykov Microsoft, @sergeybykov @msftorleans https: //github. com/dotnet/orleans/
Plan § § § § Basics Challenges of Distributed Transactions Existing Approaches Challenges Orleans transactions How they work Conclusion
Basics: Canonical Example Transfer $100 from account A to account B Atomicity Consistency Isolation Durability all or nothing constraints: no negative balances, etc. Tx 2 can’t see extra $100 in B until Tx 1 completes data is reliably stored
DBMS Takes Care of Everything … Locally Data is local No network involved Failures are mostly correlated Key focus is on performance vs. isolation/consistency tradeoff Serializable Repeatable read Read committed Read uncommitted Snapshot isolation …
Distributed Transactions Is a Different Ball Game Tx. N Pr re a ep m m Co it Begin Tx. N Prepare Commit Tx. N Commit Coordinator (DTC) Pr SQL Server Resource Managers Oracle ep ar e m m it Co Tx. N X/Open Compliant
Distributed Transactions in practice Configuration is hard CAP Compatibility is hit and miss, mostly miss Most modern vendors don’t support integration Performance is bad Latency – roundtrips with coordinator Throughput – locks and single coordinator Reliability is hard to achieve https: //blogs. msdn. microsoft. com/distributedservices/2011/11/22/troubleshooting-msdtc-communication-checklist/
Distributed Transactions Are Dead
“Life Beyond Distributed Transactions”
CQRS/ES to the Rescue
$100 A -> B Ack Bank entity Withdraw $100 Account B entity sit po De Account A entity -$100 A -> B +$100 Transfer $100 with Event Sourcing 00 $1 Transfer $100 from Account A to Account B Append-only log
CQRS + Event Sourcing No Atomicity Eventual Consistency No Isolation Durable Works well for MANY scenarios. What to do when you need more than that?
Strong Consistency Strikes Back or New Kids on The Block
Spanner
Cloud Spanner Eric Brewer, https: //static. googleusercontent. com/media/research. google. com/en//pubs/archive/45855. pdf
Cosmos DB
Cosmos DB: Consistency Options
Still A Single Database!
Microsoft Orleans
For all engineers Simple yet powerful 3 x – 10 x less code Scalable by default Programming model Orleans: Framework for the Cloud* No inherent bottlenecks No single points of failure Known best patterns and practices *In addition to cloud services, Orleans fits equally well for building a wide variety of distributed applications
Programming Model of Orleans Grain is an object with a stable ID Frontends Business Logic ACID Storage Lives forever, virtually Encapsulates state No direct external access Message passing Grain manages its own state Multiple storage systems can be used Great fit for Event Sourcing Isolated logs of state changes Excellent scalability No built-in coordination
Transfer of $100 in Orleans Persist Ack 0 10 -$ $100 A -> B k Ac $100 A -> B + k 00 Ac $1 Ack Persist Ack
What We Are Forced To Do Persist Recovery mechanism Ack 0 10 -$ $100 A -> B k Ac $100 A -> B + k 00 Ac $1 Ack Persist Record request with a unique ID to dedup Record completion Ack
Observation by Martin Kleppmann
What We Want for Transfer Operation Persist )] Ack k ] Ac d) ire qu 0 Re 10 x( + $ [T $100 A -> B [Tx(Requires. New)] $100 A -> B d ire u eq 00 R ( 1 [Tx - $ k Ac Persist Ack
We Want ACID Guarantees Persist )] Ack $500 $400 $500 $600 Ack k ] Ac d) ire qu 0 Re 10 x( + $ [T $100 A -> B [Tx(Requires. New)] $100 A -> B d ire u eq 00 R ( 1 [Tx - $ k Ac Persist Ack
That’s What Orleans Transactions Provide
Bank Grain Interface public interface IBank. Grain : IGrain. With. Integer. Key { [Transaction(Transaction. Option. Create)] Task Transfer(Guid from. Account, Guid to. Account, uint amount); }
Account Grain Interface public interface IAccount. Grain : IGrain. With. Guid. Key { [Transaction(Transaction. Option. Join)] Task Withdraw(uint amount); [Transaction(Transaction. Option. Join)] Task Deposit(uint amount); [Transaction(Transaction. Option. Create. Or. Join)] Task<uint> Get. Balance(); }
Transfer Operation public class Bank. Grain : Grain, IBank. Grain { Task Transfer(Guid from. Account, Guid to. Account, uint amount) { var from = Grain. Factory. Get. Grain<IAccount. Grain>(from. Account); var to = Grain. Factory. Get. Grain<IAccount. Grain>(to. Account); Task t 1 = from. Withdraw(amount); Task t 2 = to. Deposit(amount); return Task. When. All(t 1, t 2); } }
Account Grain: Balance State Facet public class Balance { public uint Value { get; set; } = 1000; } public class Account. Grain : Grain, IAccount. Grain { private readonly ITransactional. State<Balance> balance; public Account. Grain( [Transactional. State("balance")] ITransactional. State<Balance> balance) { this. balance = balance; } }
Account Grain: Operations Task IAccount. Grain. Deposit(uint amount) Task IAccount. Grain. Withdraw(uint amount) { { return this. balance. Perform. Update( x => x. Value -= amount); x => x. Value += amount); } } async Task<uint> IAccount. Grain. Get. Balance() { return this. balance. Perform. Read( x => x. Value); }
This Is Pretty Much It! Just need to add some configuration to silos var builder = new Silo. Host. Builder(). Use. Localhost. Clustering() …. Add. Memory. Grain. Storage. As. Default(). Use. Transactions(); var host = builder. Build(); await host. Start. Async();
How does it work?
Architecture https: //www. microsoft. com/en-us/research/wp-content/uploads/2016/10/Eldeeb. Bernstein-Transactional. Actors-MSR-TR-1. pdf
Architecture Cluster of Silos Transaction Manager Storage TM Silo Transaction Agent
2 -Phase Commit 00) 1 $ ( aw (N) r d ith epare W A: A: Pr Ack ared p Pre TC Validate(N, {A, B}, {N-5, N-2}) [Tx(Requires. New)] $100 A -> B Success Wait for N-5 and N-2 N depends on TM Committed Tx N: A(N-5) B(N-2) B: D Prep a e B: P posit( red rep $10 Ack are(N 0) ) Transaction log
What if something fails?
2 PC Abort N) e( par e Pr A: Wait for N-5 and N-2 N depends on ed par Pre TC Validate(N, {A, B}, {N-5, N-2}) [Tx(Requires. New)] $100 A -> B TM Abort Error Tx N: A(N-5) B(N-2) B: P Pre par ed rep are (N) Transaction log
Highlights 1. ACID across non-transactional stores 2. Scales to a high rate of non-overlapping transactions 3. Intuitive programming model 4. Suitable for a wide range of internet-scale scenarios
Drawbacks 1. Cascading aborts § Can happen only due to server/OS failures, which are rare 2. Single-grain transactions require validation § Only affects latency, not throughput 3. Stand-alone TM is extra operational cost 4. Centralized TM is a scalability bottleneck § Nice problem to have § Can apply some old techniques…
Throughput 8 -core VM
Batching N) e( par e Pr A: ed par e r P TC Validate({N, O, P, Q, X, Y, Z}) [Tx(Requires. New)] $100 A -> B TM Committed ({N, O, P, Q, X}) Aborted({Y, Z}) Success Tx N: A(N-5) B(N-2) B: P Pre par ed rep are (N) Transaction log
That was the beginning (Beta)
Partnership with Research Phil Bernstein Sebastian Burckhardt Christopher Meiklejohn Alejandro Tomsic
Tx Manager v 2 N) e( par e Pr A: TC ed par e r P TM Validate(N, {A, B}, {N-5, N-2}) [Tx(Requires. New)] $100 A -> B TM Committed Success Tx N: A(N-5) B(N-2) B: P Pre par ed rep are (N) Transaction log
Drawbacks of Tx Manager v 1 1. Cascading aborts § Can happen only due to server/OS failures, which are rare 2. Single-grain transactions require validation § Only affects latency, not throughput 3. Stand-alone TM is extra operational cost 4. Centralized TM is a scalability bottleneck § Nice problem to have
Best of all…
All Code Is on Git. Hub
Conclusion § There’s still space for innovation, even in decades old problems § SQL <-> No. SQL <-> Distributed Transactions § Middle-Tier stack allows to do quite a bit § While staying agnostic of storage § Open Source all the way § You are welcome to participate!
Thank you! Sergey Bykov Microsoft, @sergeybykov @msftorleans https: //github. com/dotnet/orleans/
Dmitry Vakulenko § Top contributor for 3 years § Increased Orleans performance by 2. 6 x § Rewrote Scheduler (heart of the runtime) § 30% perf gain § Simpler and better debuggable code § Key feature of the 2. 1 release
- Slides: 52