Message Driven Architecture for Massive Service Elastic Scalability
Message Driven Architecture for Massive Service Elastic Scalability, High Availability 2011. 18 박혜웅
Massive Service Think different
What we need for Massive Service? • coupled vs decoupled architecture – decoupled architecture • • • systems for removing SPOF – for All System • – – health-checking script for RDBMS/No. SQL • • Hadoop/HBase dual namenode (next version, 0. 23) My. SQL cluster or My. SQL replication( + heartbeat) or My. SQL multiple-master blocking vs non-blocking (synchronous vs asynchronous) – – • distributed coordinator for Load balancer • • distributed data cache distributed message queue blocking(synchronous): easy coding, big resources non-blocking(asynchronous): hard coding, small resources multi-thread(single-port) vs single-thread(multi-port) – advantage of single thread cheap server • • No locking, No Synchronization easy to coding 5
What we need for Massive Service? • low cost – money • • – time: • • – • hardware based vs software based commercial software vs free software development & debugging management human resouces performance tunning – Linux options • – JVM options • – stress test socket options • – Xms, Xmx, GC option the number of processes, threads (each system) • – ulimit, . . . TCP_NODELAY, SEND/RECV_BUFFERSIZE. . . RDBMS/No. SQL options 6
What we need for 칼퇴근? • experts for each technical area = DRI(Directly Responsible Individual in Apple Inc. ) – coding & interface • – DB & storage • • • – Google Protocol Buffer, Guice, Log 4 j, Slf 4, Xstream, Jackson, Java mail, . . system management • – coordinator(Zookeeper) cache server(Redis, Memcached, Ehcache) queue server(Rabbit. MQ, Zero. MQ) util software • – Map. Reduce, machine learning distributed system software • • • – Java NIO, Netty data analysis • – RDBMS(My. SQL, My. Batis) No. SQL(Hbase) storage(DAS, NAS, HDFS, Haystack. . . ) network & threading • – code convention, design pattern, UML Linux, monitoring tools, JMX hardware • L 4 switch 7
What we need for 칼퇴근? • fast & easy development/debugging – good architecture • • • – common util classes • – JUnit well-known system or not? • • Apache Commons, Google Guava, . . . Test Driven Development (TDD) • – system architecture design pattern code convention RDBMS vs No. SQL JSON vs Google Protocol Buffer JUnit vs Guice easy management – logging system • – – logging, collecting, parsing, log visualization JMX Admin/Monitoring tools or web pages 8
many Kinds of Decoupling • decoupling(removing) of SPOF and our system – Distributed Coordinator process Coordinator SPOF process • decoupling of business logic and data – Distributed Cache process logic process data logic DB Cache data process logic process DB data logic data • decoupling of function and control(message) – Message Queue process function 9 process Queue process function message function
the steps of Decoupling (step 1) • Distributed Coordinator – registry: important data (small size) • • • – server status server configuration common data removing SPOF from our system Coordinator registry process function data registry DB DB process function data registry 10
the steps of Decoupling (step 2) • Distributed Data Cache – fast read/write in memory • – alleviate DB overload • • – – – 10~100 times faster than DB query. read query: read cache instead of DB. write query: lazy update for DB with write-through queue. remove duplicated data remove overhead of data synchronization among processes. fault tolerant system • no matter what process terminated in the same cluster. Coordinator registry process function data process function Cache data DB data cluster data process function data function process function DB data 11 function
the steps of Decoupling (step 3) • Distributed Message Queue – scale out (elastic scalibility) • • – fault tolerant system • – but lazy processing system monitoring • Coordinator registry when all process terminated, message queue server preserves messages. prevent server overload or failure. • – auto scaling by fan-out exchange rule. light-weight processes(daemons). just monitor queue status. process function Coordinator registry function Cache data data process Queue process function message function cluster process function DB data 12 process Queue process function message function
Scale Out cluster Coordinator registry node Cache data node cluster node Cache cluster data node node process task #1 Queue message function n connections function node Queue task #2 message node DB data 13 message work queue process function node
SEDA vs Message Driven Architecture process data/heap area global variable thread Queue thread function event function SEDA thread data DB data service node node process Queue process function message function Cache data node DB data node process Queue process function message function 14 Coordinator registry MDA
code of Message Driven Architecture • simple chatting service (simple client-server based model vs MDA) /** Simple Client-Server Model **/ /* Send Thread */ my. Info = xml. get. Info(xml. File); // from local file db. set. Alive(my. Info); // updates server status /** Message Driven Architecture **/ /* Send Thread (Process) */ my. Info = Zookeeper. get. Info(zookeeper. List, my. Ip, my. Port); Zookeeper. set. Alive(my. Info); servers = connect. All(relay. Servers); //connects to other servers. queue = Queue. get. Queue(my. Info. queue); cache = Cache. get. Cache(my. Info. cache); while( (input=client. get. Input()) !=null ){ room. Info = local. Data. get. Room. Info(client. user. Id); for( user. Id: room. Info. get. User. Ids() ){ for( server : servers ){ if( server. has. User(user. Id) ) server. send(user. Id, input); } } } while( (input=client. get. Input()) !=null ){ room. Info = cache. get. Room. Info(client. user. Id); for( user. Id : cache. get. User. Ids(room. Info. no) ){ queue. publish(new Message(user. Id, input)); } } /* Receive Thread */ while(true){ message = socket. receive(); // from other server user = local. Data. get. User(message. user. Id); //from local client = get. Client(message. user. Id); client. send(user. name + ": " + message. input); } /* Receive Thread (Process) */ while(true){ message = queue. consume(); // from queue user = cache. get. User(message. user. Id); // from cache client = get. Client(message. user. Id); client. send(user. name + ": " + message. input); } inter-server networking (p 2 p) queueing/dequeuing (work queue) 15
Appendix Think deeply
Single-thread vs Multi-thread • Multi-thread – I/O intensive task (blocked task) process thread DB thread data • Single-thread – CPU/Mem intensive task (non-blocked task) process thread data process thread MEM data process thread data 19
- Slides: 19