Network Serialization and Routing in World of Warcraft

  • Slides: 61
Download presentation
Network Serialization and Routing in World of Warcraft Joe Rumsey jrumsey@blizzard. com Twitter: @joerumz

Network Serialization and Routing in World of Warcraft Joe Rumsey [email protected] com Twitter: @joerumz

What is JAM? J A M oe’s utomated essages

What is JAM? J A M oe’s utomated essages

The Problem Game servers need to communicate with each other

The Problem Game servers need to communicate with each other

Manual serialization is error-prone void Serialize(stream &msg) { vector<int> values; //. . . Fill

Manual serialization is error-prone void Serialize(stream &msg) { vector<int> values; //. . . Fill in some values. . . msg << values. size(); for(int i = values. size(); --i; ) { msg << values[i]; } } void Deserialize(stream &msg) { vector<int> values; int size; msg >> size; values. resize(size); for(int i = size; i--; ) { msg >> values[i]; } }

Manual serialization doesn’t scale World Of Checkers --server boundary--

Manual serialization doesn’t scale World Of Checkers --server boundary--

Goals • DRY - Don’t Repeat Yourself • Eliminate boilerplate to reduce bugs •

Goals • DRY - Don’t Repeat Yourself • Eliminate boilerplate to reduce bugs • No more hand-coded serialize/deserialize • Spend more time on the game, not the protocol • Build a helpful robot that writes our code for us

Goal: Human readable code struct Checker. Captured { Checker. ID id; Checker. ID captured.

Goal: Human readable code struct Checker. Captured { Checker. ID id; Checker. ID captured. By; u 8 jump. Type; }; void Capture(Checker. ID id, Checker. ID by, JUMP_TYPE jump. Type) { Checker. Captured msg; msg. id = id; msg. captured. By = by; msg. jump. Type = jump. Type; Send(&msg); }

Implementation Details

Implementation Details

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages • Receive messages • Configure routing info

1 -to-1 mapping of. jam messages to C++ classes // From Checkers. jam message

1 -to-1 mapping of. jam messages to C++ classes // From Checkers. jam message Checker. Capture. Credit { Checker. ID captured. Checker. ID; Checker. ID captured. By; u 8 jump. Type; }; // 100% Generated code in Jam. Checkers. cpp class Checker. Capture. Credit : public Jam. Message { public: // Message decoders BOOL Get(Binary. Decoder &decoder); BOOL Get(JSONDecoder &decoder); // Message encoders BOOL Put(Binary. Encoder &encoder) const; BOOL Put(JSONEncoder &encoder) const; /**** DATA START ****/ Checker. ID captured. Checker. ID; Checker. ID captured. By; u 8 jump. Type; /**** DATA STOP ****/ // Lots more stuff. . . };

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages • Receive messages • Configure routing info

Auto-generated serialization code //NOTICE: This is generated code. DO NOT EDIT! BOOL Checker. Capture.

Auto-generated serialization code //NOTICE: This is generated code. DO NOT EDIT! BOOL Checker. Capture. Credit: : Put(Binary. Encoder &_encoder) const { _encoder. Begin. Message(CODE, NAME); _encoder. Put("captured. Checker. ID", captured. Checker. ID); _encoder. Put("captured. By", captured. By); _encoder. Put("jump. Type", jump. Type); _encoder. End. Message(CODE, NAME); return TRUE; }

Flex and Bison make writing parsers easy Other tools • ANTLR • GOLD •

Flex and Bison make writing parsers easy Other tools • ANTLR • GOLD • PLY (Python Lex & Yacc) • Boost. Spirit Flex & Bison - parser generators

JAM File syntax is described to Bison part of jam. y

JAM File syntax is described to Bison part of jam. y

From. jam to. cpp 2004 2013

From. jam to. cpp 2004 2013

TRL Turns. jam into C++ {@ define Output. Message(msg, encoders, decoders) @} // //

TRL Turns. jam into C++ {@ define Output. Message(msg, encoders, decoders) @} // // NOTICE: This is generated code. DO NOT EDIT! // class {{ msg. struct. Name }} : public Jam. Message { public: static u 32 CRC; static u 16 CODE; static cchar *NAME; // No argument constructor: {{ msg. struct. Name }}() { {@ foreach f in msg. fields @} {@ if f. has. Default @} {{ f. name }} = {{ f. def. Value }}; {@ end if @} {@ end foreach @} } TRL to generate a message class definition See Also • CTemplate • ng. Template • Django (HTML focused) • Jinja (Python)

Fill out a dictionary and feed it to TRL {@ foreach f in msg.

Fill out a dictionary and feed it to TRL {@ foreach f in msg. fields @} {@ if f. has. Default @} {{ f. name }} = {{ f. def. Value }}; {@ end if @} {@ end foreach @}

Global Feature addition using TRL TODO: Figure out how to illustrate this. Looking through

Global Feature addition using TRL TODO: Figure out how to illustrate this. Looking through history of our TRL files and jamgen would be useful. In fact, a screenshot of a diff of one of those changes would work well here.

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages • Receive messages • Configure routing info

Create a message, fill in data, call send void Checker: : On. Captured(Checker. ID

Create a message, fill in data, call send void Checker: : On. Captured(Checker. ID captured. By, JUMP_TYPE how) { Checker. Captured. Credit msg; msg. captured. Checker. ID = Get. ID(); msg. captured. By = captured. By; msg. jump. Type = how; Jam. ID destination = Get. Router()->Get. Credit. Manager. ID(); Get. Router()->Send(destination, &msg); }

Structs and arrays in messages message Group. Update { Group. ID group; array<. Checker.

Structs and arrays in messages message Group. Update { Group. ID group; array<. Checker. ID> checkers; }; /*** DATA START ***/ Group. ID group; vector<Checker. ID> checkers; /*** DATA STOP ***/ void Group. Service: : Send. Update(Group. ID id) { Group. Update msg; msg. group = id; msg. checkers. resize(MAX_GROUP_SIZE); //. . . }

Definitions • Message - serialized structure defined in a. jam file • Protocol -

Definitions • Message - serialized structure defined in a. jam file • Protocol - a collection of messages • Service - a module of code that implements message handlers for one or more protocols • Program - can be composed of multiple services

Message Destinations void Match. Service: : Create. Board(u 64 width, u 64 height) {

Message Destinations void Match. Service: : Create. Board(u 64 width, u 64 height) { Board. ID = Generate. Board(); // Send to a known, connected, service m_p. Server->Send(m_board. Server. ID, &msg); } void Match. Service: : Game. Over(u 32 game. ID, u 64 winner. ID) { msg. game. ID = game. ID; msg. winner = winner. ID(); // Send to a service type, non-specified ID m_p. Server->Send(JAM_SERVER_STATS_TRACKER, &msg); m_p. Server->Broadcast(JAM_SERVER_STATS_TRACKER, &msg); } void Checker: : Heal. Checker(Checker. ID to. Heal, u 32 amount) { Checker. Heal msg; msg. healed. By = Get. ID(); msg. amount = amount; // Send a message to a specific object m_p. Server->Send(to. Heal, &msg); }

Message routing by type Matchmaker. Add. Player add. Msg; add. Msg. player = Get.

Message routing by type Matchmaker. Add. Player add. Msg; add. Msg. player = Get. Player. ID(); add. Msg. rank = Get. Rank(); // No Jam. ID needed, send to any Matchmaker // May be queued until a Matchmaker is available m_p. Service->Send(JAM_SERVER_MATCHMAKER, &add. Msg);

Send a message and expect a response Matchmaker. Add. Player add. Msg; add. Msg.

Send a message and expect a response Matchmaker. Add. Player add. Msg; add. Msg. player = Get. Player. ID(); add. Msg. level = Get. Level(); // Send to any Matchmaker, Player. Added. Handler // will be called with response when complete m_p. Service->Send. Registered<Player. Added>( JAM_SERVER_MATCHMAKER, &add. Msg );

Send a message to an object void Checker. Group: : Change. Boards(u 32 new.

Send a message to an object void Checker. Group: : Change. Boards(u 32 new. Board) { Checker. Change. Board msg; msg. board. ID = new. Board; for(int i = 0; i < m_checkers. size(); i++) { m_p. Server->Send(m_checkers[i]->Get. ID(), &msg); } }

Each object is owned by one server class Checker { //. . . Checker.

Each object is owned by one server class Checker { //. . . Checker. ID m_id; Jam. ID m_server. ID; Jam. ID Get. Server() { return m_server. ID; } Checker. ID Get. ID() { return m_id; } //. . . };

How messages get routed void Board. Server: : Send(Checker *p. Checker, Jam. Message *p.

How messages get routed void Board. Server: : Send(Checker *p. Checker, Jam. Message *p. Message) { m_p. Jam. Server->Send(p. Checker->Get. Server(), p. Checker->Get. ID(), p. Message); }

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages • Receive messages • Configure routing info

On receipt, look up and dispatch // static callback registered with JAM by protocol

On receipt, look up and dispatch // static callback registered with JAM by protocol ID // called for each incoming message void Board. Server: : Checker. Dispatch(Jam. Link &link, Jam. Message *p. Message) { Checker. ID dest. ID = p. Message->Get. Destination(); Checker *p. Checker = Get. Checker. Object(dest. ID); p. Checker->Queue. Message(p. Message); switch(p. Message->Get. Protocol. CRC()) { case JAMChecker. Protocol_CRC: Jam. Checker. Protocol: : Dispatch<Checker>(p. Message, p. Checker); } }

Jam. Link &link, void Board. Server: : Checker. Dispatch( Jam. Message *p. Message) {

Jam. Link &link, void Board. Server: : Checker. Dispatch( Jam. Message *p. Message) {

Generated Dispatch methods //NOTICE: This is generated code. DO NOT EDIT! template<typename HANDLER_T> static

Generated Dispatch methods //NOTICE: This is generated code. DO NOT EDIT! template<typename HANDLER_T> static JAM_RESULT Dispatch(Jam. Message *p. Message, HANDLER_T *p. Handler) { switch(p. Message->Get. Code()) { case JAM_MSG_Checker. Heal: result = p. Handler->Checker. Heal. Handler(link, (Checker. Heal *)p. Message); break; // cases for rest of protocol's messages. . .

Generated message handler prototypes // A message handler prototype is auto-generated for each message

Generated message handler prototypes // A message handler prototype is auto-generated for each message // in the protocol. #include these declarations in the middle // of your hand constructed class. JAM_RESULT Checker. Heal. Handler(Jam. Link &link, Checker. Heal *msg); JAM_RESULT Checker. Damage. Handler(Jam. Link &link, Checker. Damage *msg); JAM_RESULT Checker. Powerup. Handler(Jam. Link &link, Checker. Powerup *msg); JAM_RESULT Checker. King. Handler(Jam. Link &link, Checker. King *msg); #include this in the middle of a class

Message handler methods JAM_RESULT Checker: : Checker. Heal. Handler(Checker. Heal *p. Message) { m_health

Message handler methods JAM_RESULT Checker: : Checker. Heal. Handler(Checker. Heal *p. Message) { m_health += p. Message->amount; LOG("Checker %d was healed for %d by checker %d", Get. ID(), p. Message->amount, p. Message->healed. By); return JAM_OK; }

Send and Receive void Checker: : Heal. Checker(Checker. ID to. Heal, u 32 amount)

Send and Receive void Checker: : Heal. Checker(Checker. ID to. Heal, u 32 amount) { Checker. Heal msg; msg. healed. By = Get. ID(); msg. amount = amount; // Send a message to a specific object m_p. Server->Send(to. Heal, &msg); } JAM_RESULT Checker: : Checker. Heal. Handler(Checker. Heal *p. Message) { m_health += p. Message->amount; LOG("Checker %d was healed for %d by checker %d", Get. ID(), p. Message->amount, p. Message->healed. By); return JAM_OK; }

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages

Development Cycle • Describe the protocol • Generate serialization and dispatch • Send messages • Receive messages • Configure routing info

Define services void Matchmaker: : Configure(Jam. Server *p. Server) { Jam. Route. Config &route.

Define services void Matchmaker: : Configure(Jam. Server *p. Server) { Jam. Route. Config &route. Config = p. Server->Get. Route. Config(); route. Configure. Inbound<Matchmaker. Protocol>( this, Matchmaker: : Dispatch. Message); route. Configure. Outbound<Matchmaker. Response. Protocol>(); } Configure protocols the Matchmaker service sends and receives

Route. Config maintains a protocol to handler mapping

Route. Config maintains a protocol to handler mapping

Handlers have access to sender and other metadata about received messages JAM_RESULT Board. Server:

Handlers have access to sender and other metadata about received messages JAM_RESULT Board. Server: : Add. Player. Handler(Jam. Link &link, Add. Player *msg) { LOG("Adding player %s from server %s", IDSTR(msg->player. ID), link. Describe(). c_str()); // Do stuff return JAM_OK; }

Coarse and fine-grained queueing and locking strategies are available Race Condition

Coarse and fine-grained queueing and locking strategies are available Race Condition

Receiving via Message Queue void Matchmaker: : Configure() { // Messages received at any

Receiving via Message Queue void Matchmaker: : Configure() { // Messages received at any time are placed into a queue route. Configure. Inbound<Matchmaker. Protocol>( this, &m_message. Queue); } void Matchmaker: : Idle() { // Queue is processed in one thread at a known time p. Server->Process. Queue(&m_message. Queue, this); }

Global lock dispatching

Global lock dispatching

Raw concurrent handlers

Raw concurrent handlers

Lock Policies class Matchmaker. Lock. Policy { Matchmaker *m_owner; void Lock(Jam. Message *msg, Jam.

Lock Policies class Matchmaker. Lock. Policy { Matchmaker *m_owner; void Lock(Jam. Message *msg, Jam. Message. Queue **pp. Queue) { // Adding a player requires a write lock if(msg->Get. Code() == JAM_MSG_Matchmaker. Add. Player) { m_owner->Acquire. Write. Lock(); } else { m_owner->Acquire. Read. Lock(); } } void Unlock(Jam. Message *msg) { /* Same logic, release lock */ } }

Incoming messages are refcounted • Message passed to handler is a refcounted object •

Incoming messages are refcounted • Message passed to handler is a refcounted object • Possible to retain a message pointer until later • Smart pointers are available • Messages contain no pointers to any other objects • No circular references are possible

CPU And Bandwidth Efficiency

CPU And Bandwidth Efficiency

JAM is either efficient or backwards compatible 2004 - Assumed binary compatibility

JAM is either efficient or backwards compatible 2004 - Assumed binary compatibility

Negotiation means dead-simple binary serialization most of the time

Negotiation means dead-simple binary serialization most of the time

In some cases, can just memcpy it onto the wire // This message could

In some cases, can just memcpy it onto the wire // This message could easily be memcpy'ed onto the wire class Create. Checker : public Jam. Message { /**** DATA START ****/ u 32 checker. Type; u 32 owner; /**** DATA STOP ****/ // Code. . . };

Generated code means easy optimizations _encoder. Put("captured. Checker. ID", captured. Checker. ID); _encoder. Put("captured.

Generated code means easy optimizations _encoder. Put("captured. Checker. ID", captured. Checker. ID); _encoder. Put("captured. By", captured. By); _encoder. Put("jump. Type", jump. Type);

Fallback to JSON Serialization • Switch to JSON serialization when binary CRC check fail

Fallback to JSON Serialization • Switch to JSON serialization when binary CRC check fail • Great for programmers • Way more expensive (CPU and Bandwidth) • Never allowed on public facing protocols • Even internally it’s sometimes unreasonable

JSON { "_msg. ID": 10, "type": 6, "error": 0, "desc": { "m_id": "T 2

JSON { "_msg. ID": 10, "type": 6, "error": 0, "desc": { "m_id": "T 2 R 00 S 40. 00 E 14815726 P 10987 H 127. 0. 0. 1: 14001", "m_host": "127. 0. 0. 1", "m_partition. ID": 0, "m_config. ID": 0, "m_build. Num": 0, "m_type": 40, "m_sub. Type": 0 } }

Protocol Negotiation

Protocol Negotiation

Message overhead

Message overhead

JSON vs. Binary performance

JSON vs. Binary performance

Still highly successful - some network tools run on old versions frequently

Still highly successful - some network tools run on old versions frequently

Facebook's Thrift are good alternatives

Facebook's Thrift are good alternatives

For both speed AND inter-version compatibility, there are better choices

For both speed AND inter-version compatibility, there are better choices

protobufs sometimes wins on bandwidth, but JAM is faster

protobufs sometimes wins on bandwidth, but JAM is faster

Writing our own gives us ultimate control over everything • Automated serialization • Easy

Writing our own gives us ultimate control over everything • Automated serialization • Easy yet flexible message dispatching • High performance • Inter-version compatibility • Less tedium = more awesome

Thanks! Questions? Joe Rumsey jrumsey@blizzard. com Twitter: @joerumz Disclaimer: Blizzard is not really making

Thanks! Questions? Joe Rumsey [email protected] com Twitter: @joerumz Disclaimer: Blizzard is not really making World of Checkers