Ne ST Network Storage Flexible Commodity Storage Appliances
Ne. ST: Network Storage Flexible Commodity Storage Appliances John Bent, Venkateshwaran Venkataramani, Miron Livny, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau University of Wisconsin, Madison
Ne. ST: Network Storage Flexible, commodity based, software-only storage appliances Building a Ne. ST should be as easy as: Finding a networked machine “Dropping” some software on it Self-configuring to best utilize machine www. nestproject. com
Ne. ST resource management Dynamic user account creation provides storage for migratory grid users Storage reservations and quota systems intelligent scheduling of data intensive applications Matchmaking match storage opportunities & storage consumers www. nestproject. com
Condor Ne. STs Better, smarter checkpoint servers Checkpoints are just another data file Data replicated/migrates to different Ne. STs Condor jobs access data from closest Ne. ST Flexible policy support for managing disk and network resources www. nestproject. com
New worlds, New problems Diverse hardware, software platforms - Netapp, EMC advantage • fewer platforms, control over OS - Our approach • Automate configuration to each host system – H/W (Disks): use file system or self-manage – S/W (File System): use either read/write or mmap Implication: Flexibility at a cost? Key is design of the software www. nestproject. com
Ne. ST structure Modules for communication, transfer & storage Protocol layer • Pluggable protocols allow diverse protocols to be mapped into common control flows Transfer layer • Different concurrency architectures to maximize system throughput on diverse platforms Storage layer • Provides abstract interface to disks www. nestproject. com
Ne. ST Structure GFTP Ne. ST Wi. ND HTTP NFS Protocol Layer Concurrency Architecture Multi. Nonblocking process threaded Storage Layer Raw disk Local FS RAID Control Logic Memory www. nestproject. com
Many Protocols, Single Server Single point of control - Storage quotas/guarantees can be supported - Bandwidth can be controlled & Qo. S provided Single administrative interface - Set policies, Manage user accounts Maintainable S/W - Shared code base reduces replication, increases maintainability www. nestproject. com
Protocol layer implementation Each protocol listens on well-defined port Central control accepts connections Protocol layer reads from connection and returns generic request object Analagous to Linux V-nodes – Add new protocol by writing a couple of methods www. nestproject. com
Virtual protocol class nest. Protocol { public: virtual nest. Request* receive. Request( fd_set *ready ) = 0; virtual ssize_t Receive( char *buffer, ssize_t n )= 0; virtual ssize_t Send( char *buffer, ssize_t n ) = 0; virtual bool send. Ack( ) = 0; virtual bool send. Error. Message( int error ) = 0; virtual bool send. Directory. List( Nest. File. Info *list ); virtual bool send. Pwd( char *pwd ); virtual bool send. Transfer. Info( class nest. Request *req ); virtual bool send. Filesize( int retval, long size) = 0; virtual bool send. Quota. Info( struct User. Usage* usage ); virtual bool has. Socket() = 0; virtual int get. Socket() = 0; }; www. nestproject. com
Virtual protocol class Plus some static methods static int listen( int port ); static nest. Protocol* accept( int socket, sockaddr_in *address ); static Nest. Reply. Status initiate. Third. Party( nest. Client *client, nest. Protocol **connection ); www. nestproject. com
Grid FTP virtual protocol Good news? – Hard work already done – Already implemented libraries have support for parallel streams, authentication, etc. Bad news? – Slight mismatch to integrate callback model into synchronous API www. nestproject. com
Grid FTP - listen // static initialization routine int nest. Protocol. Grid. Ftp: : listen( int port ) { globus_result_t result; globus_ftp_control_server_t *server; globus_module_activate( GLOBUS_FTP_CONTROL_MODULE ); server = (globus_ftp_control_server_t *) Malloc( sizeof( struct globus_ftp_control_server_s ) ); result = globus_ftp_control_server_handle_init( server ); if ( result != GLOBUS_SUCCESS ) { goto ERROR; } short unsigned Port = (short unsigned) port; result = globus_ftp_control_server_listen( server, &Port, listen. Callback, NULL ); if ( result != GLOBUS_SUCCESS ) { goto ERROR; } if ( Spipe( pipes ) < 0 ) { goto ERROR; } } www. nestproject. com
Grid FTP - receive. Request nest. Request* nest. Protocol. Grid. Ftp: : receive. Request( fd_set *ready ){ // ignore the fd_set char msg[MESSAGESIZE]; snprintf( msg, MESSAGESIZE, "%s", "Gftp receive. Request called: " ); nest. Request *req = this->request; if ( req != NULL ) { snprintf( &(msg[strlen(msg)]), MESSAGESIZE - strlen(msg), "%sn", "Actually have a request to give. n" ); } else { snprintf( &(msg[strlen(msg)]), MESSAGESIZE - strlen(msg), "%sn", "No request to give. n" ); } nest_debug( 1, msg ); return req; } www. nestproject. com
Grid FTP - send. Filesize bool nest. Protocol. Grid. Ftp: : send. Filesize( int retval, long size){ assert( req. Type == FILESIZE_REQUEST ); switch( retval ) { case NEST_SUCCESS: snprintf( buffer, FILEBUFFER, "213 %ldrn", size ); break; case NEST_REMOTE_FILE_NOT_FOUND: snprintf( buffer, FILEBUFFER, "550 File not foundrn" ); break; default: assert( 0 ); } return send. Message( buffer ); } www. nestproject. com
Grid FTP - send. Message bool nest. Protocol. Grid. Ftp: : send. Message( const void *vptr ) { globus_result_t result; wait. Idle(); set. Wait( true ); result = globus_ftp_control_send_response( handle, (char *)vptr, command. Callback, this ); if ( result != GLOBUS_SUCCESS ) { fprintf( stderr, "globus_ftp_control_send_response error: %sn", result. To. String( result ) ); free( handle ); return false; } return true; } www. nestproject. com
Grid FTP - command. Callback void nest. Protocol. Grid. Ftp: : command. Callback( void * arg, struct globus_ftp_control_handle_s * handle, globus_object_t * error) { nest_debug( 1, "command_callback called - set wait falsen" ); if( error != GLOBUS_SUCCESS ) { fprintf(stderr, ">> command_callback: error %sn", globus_object_printable_to_string( error ) ); return; } // set the wait flag to false nest. Protocol. Grid. Ftp *connection = (nest. Protocol. Grid. Ftp *)arg; connection->set. Wait( false ); } www. nestproject. com
Concurrency architecture Three difficult goals – Low latency – High bandwidth – Multiple simultaneous clients No single portable solution – Multiple models provide solutions on a range of different platforms • Multi-threaded, multi-process, single process nonblocking www. nestproject. com
Flexible Concurrency Nonblocking Multi-process Multi-threaded Concurrency architecture Control logic creates transfer object – Virtual connection from the protocol layer – Virtual connection from the storage layer – Transfer object passed to concurrency layer – Control logic informed when transfer ends www. nestproject. com
Concurrency Models Nonblocking model – Performs only non-blocking I/O – Selects on file-descriptors / sockets from each transfer object Pre-allocated pool of processes – Descriptors are passed over a pipe – Transfer object recreated on other side – Each process does a blocking transfer Pre-allocated pool of threads – Transfer object enqueued by server – Dequeued by an available thread www. nestproject. com
Concurrency Models and GFTP Nonblocking model – Not yet supported. – Grid FTP libraries do not expose socket on which to select. Pre-allocated pool of processes – Not yet supported. – Haven’t figured out how to send a Grid. FTP connection over a pipe Pre-allocated pool of threads – No problem. Fully integrated. Other models could work. . . www. nestproject. com
Storage Layer Three needed areas of flexibility – File systems interfaces • Example: read()/write() or mmap() – Abstract storage models • RAID, JBOD, etc. • Memory storage model also a possibility – Provide file system interface to remote memory – Useful for buffering systems like Kangaroo • Virtual storage model akin to virtual protocol layer – User account administration • Creation and removal • Quotas and guarantees for users and groups www. nestproject. com
- Slides: 22