Grid Programming in Python with py Globus Keith

Grid Programming in Python with py. Globus Keith R. Jackson Distributed Systems Department Lawrence Berkeley National Lab Grid Programming in Python

Overview • Python Co. G Kit Overview / Goals • Status • Module Overview — Job Submission — Data Transfer • Grid. FTP • Gass. Copy — Remote File IO — Secure Network IO — GSI Soap • Future Plans • Contacts & Acknowledgements Grid Programming in Python

Python Co. G Kit • The Python Co. G Kit provides a mapping between Python and the Globus Toolkit. It extends the use of Globus by enabling access to advanced Python features such as events and objects for Grid programming. • Hides much of the complexity of Grid programming behind simple object-oriented interfaces. • The Python Co. G Kit is implemented as a series of Python extension modules that wrap the Globus C code. • Uses SWIG (http: //www. swig. org) to help generate the interfaces. Grid Programming in Python

Python Co. G Kit Goals • Supports rapid development of high-level Grid services, while providing access to the underlying performance of the Globus toolkit. • Develop a common set of reusable components in Python for accessing Grid services. • Focus on supporting the development of Science Portals, Problem Solving Environments, and science applications that access Grid resources. Grid Programming in Python

Python Co. G Kit Basics • Everything is contained in the py. Globus package. • The low-level C-wrappers are all named modulec, e. g. , ftp. Clientc, gram. Clientc — Shouldn’t need to access these directly, instead use the Python wrappers (ftp. Client, gram. Client) • Exceptions are used to handle error conditions — py. Globus. util. Globus. Exception is the base class for all py. Globus exceptions. It extends the base Python exceptions. Exception class • The Python wrappers classes manage the underlying memory & module activation/deactivation Grid Programming in Python

Status: Python Co. G Kit • Basic services are provided accessing: — — — — — Security (security) Remote job submission and monitoring (gram. Client) Secure high-performance network IO (io) Protocol independent data transfers (gass. Copy) High performance Grid FTP transfers (ftp. Client) Support for building Grid FTP servers (ftp. Control) Remote file IO (gass. File) GASS Cache management (gass. Cache) Replica management (replica. Management) Replica Catalog (replica. Catalog) Grid Programming in Python

gram. Client Example from py. Globus. gram. Client import * from threading import * cond = 0 rm = “host. lbl. gov” rsl = “&(executable=/bin/date)” def main(rm, rsl): cond. V = Condition(Lock()) try: g. C = Gram. Client() cb. Contact = def done(cv, contact, state, err): global cond if state == JOB_STATE_FAILED: print "Job failed" elif state == JOB_STATE_DONE: print "Job is done“ else: print “ERROR: “, err cv. acquire() cond = 1 cv. notify() cv. release() g. C. set_callback(done, cond. V) job. Contact = g. C. submit_request(rm, rsl, JOB_STATE_ALL, cb. Contact) while cond == 0: cond. V. acquire() cond. V. wait() cond. V. release() g. C. remove_callback(cb. Contact) except Gram. Client. Exception, ex: print ex Grid Programming in Python

C Job Submission Example callback_func(void *user_arg, char *job_contact, globus_l_globusrun_gramrun(char * request_string, int state, int errorcode) unsigned long options, { char *rm_contact) globus_i_globusrun_gram_monitor_t *monitor; { monitor = (globus_i_globusrun_gram_monitor_t *) user_arg; char *callback_contact = GLOBUS_NULL; globus_mutex_lock(&monitor->mutex); char *job_contact = GLOBUS_NULL; monitor->job_state = state; globus_i_globusrun_gram_monitor_t monitor; switch(state) int err; { monitor. done = GLOBUS_FALSE; case GLOBUS_GRAM_PROTOCOL_JOB_STATE_PENDING: monitor. verbose=verbose; { globus_mutex_init(&monitor. mutex, GLOBUS_NULL); globus_i_globusrun_gram_monitor_t *monitor; globus_cond_init(&monitor. cond, GLOBUS_NULL); monitor = (globus_i_globusrun_gram_monitor_t *) user_arg; globus_mutex_lock(&monitor->mutex); err = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE); monitor->job_state = state; if(err != GLOBUS_SUCCESS) switch(state) { … } { err = globus_gram_client_callback_allow( case GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED: globus_l_globusrun_gram_callback_func, if(monitor->verbose) (void *) &monitor, { globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILEDn"); &callback_contact); if(err != GLOBUS_SUCCESS) } { … } monitor->done = GLOBUS_TRUE; err = globus_gram_client_job_request(rm_contact, break; request_string, GLOBUS_GRAM_PROTOCOL_JOB_STATE_ALL, case GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE: callback_contact, &job_contact); if(monitor->verbose) { globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONEn"); } monitor->done = GLOBUS_TRUE; if(err != GLOBUS_SUCCESS) { … } globus_mutex_lock(&monitor. mutex); while(!monitor. done) break; } } globus_cond_signal(&monitor->cond); globus_mutex_unlock(&monitor->mutex); } { globus_cond_wait(&monitor. cond, &monitor. mutex); globus_mutex_unlock(&monitor. mutex); globus_gram_client_callback_disallow(callback_contact); Grid Programming in Python globus_free(callback_contact);

Data Transfer • Provides both protocol dependent and protocol independent transfer mechanisms — Directly use the Grid FTP protocol to transfer the data. • • Uses GSI for mutual authentication. Supports partial file transfers and restarts. Supports tunable network parameters. Provides extensions to standard FTP protocol to support striped and parallel data transfer. — Use the gass. Copy module for protocol independent transfers. • It currently supports the ftp, gsiftp, http, and https protocols. Grid Programming in Python

ftp. Client Example from py. Globus import ftp. Client handle. Attr = ftp. Client. Ftp. Client. Handle. Attr() op. Attr = ftp. Client. Ftp. Client. Operation. Attr() marker = ftp. Client. Restart. Marker. Ftp. Client. Restart. Marker() f. Client = ftp. Client. Ftp. Client(handle. Attr) f. Client. get(url, op. Attr, marker, done_func, cond. V) handle = f. Client. register_read(buf, data_func, 0) def data_func(cv, handle, buffer, buf. Handle, buf. Len, offset, eof, error): g_dest. write(buffer) if not eof: try: handle = f. Client. register_read(g_buffer, data_func, 0) except Exception, e: Grid Programming in Python

Performance Options in ftp. Client • Setting tcpbuffer size from py. Globus import ftp. Control battr = ftp. Control. Tcp. Buffer() battr. set_fixed(64*1024) Or battr. set_automatic(16*1024, 8*1024, 64*1024) op. Attr. set_tcp_buffer(battr) • Setting parallelism para = ftp. Control. Parallelism() para. set_mode(ftp. Control. PARALLELISM_FIXED) para. set_size(3) op. Attr. set_parallelism(para) Grid Programming in Python

Using ftp. Client plugins • Debug Plugin from py. Globus import ftp. Client f = open(“/tmp/out”, “w+”) debug. Plugin = ftp. Client. Debug. Plugin(f, “foobar”) handle. Attr. add_plugin(debug. Plugin) • Restart. Marker Plugin restart. Plugin = ftp. Client. Restart. Marker. Plugin(begin. CB, marker. CB, done. CB, None) Grid Programming in Python

gass. Copy Example from py. Globus import gass. Copy src. Attr = gass. Copy. Gass. Copy. Attr() handle. Attr = gass. Copy. Gass. Copy. Handle. Attr() dest. Attr = gass. Copy. Gass. Copy. Attr() ftp. Src. Attr = gass. Copy. Ftp. Operation. Attr() ftp. Dest. Attr = gass. Copy. Ftp. Operation. Attr() src. Attr. set_ftp(ftp. Src. Attr) dest. Attr. set_ftp(ftp. Dest. Attr) copy = gass. Copy. Gass. Copy(handle. Attr) copy_url_to_url(src. Url, src. Attr, dest. Url, dest. Attr) Grid Programming in Python

Remote File IO • Supports reading and writing remote files from http, https, ftp, gsiftp servers. — Can be opened either as normal Python file objects or as int file descriptors suitable for use with the os module. from py. Globus import gass. File = gass. File. Gass. File() f = gass. File. fopen(“gsiftp: //foo. bar. com/file”, “r”) lines = f. readlines() gass. File. fclose(f) Grid Programming in Python

Secure High-Performance IO • Uses the Grid Security Infrastructure to provide authentication. • Provides access to the underlying network parameters for tuning performance. • Integrates with the Python networking model Grid Programming in Python

GSI TCP Server Example from py. Globus import io attr = io. TCPIOAttr() attr. set_authentication_mode( io. GLOBUS_IO_SECURE_AUTHENTICATION_MODE_GSS_API) auth. Data = io. Auth. Data() auth. Data. set_callback(auth_callback, None) attr. set_authorization_mode( io. GLOBUS_IO_SECURE_AUTHORIZATION_MODE_CALLBACK, auth. Data) attr. set_channel_mode( io. GLOBUS_IO_SECURE_CHANNEL_MODE_GSI_WRAP) soc = io. GSITCPSocket() port = soc. create_listener(attr) soc. listen() child. Soc = soc. accept(attr) buf = Buffer(size) bytes. Read = child. Soc. read(buf, size) Grid Programming in Python

GSI TCP Client Example from py. Globus import io attr = io. TCPIOAttr() attr. set_authentication_mode( io. GLOBUS_IO_SECURE_AUTHENTICATION_MODE_G SS_AP) auth. Data = io. Auth. Data() attr. set_authorization_mode( io. GLOBUS_IO_SECURE_AUTHORIZATION_MODE_SE LF, auth. Data) attr. set_channel_mode( io. GLOBUS_IO_SECURE_CHANNEL_MODE_GSI_WRA P) soc = io. GSITCPSocket() soc. connect(host, port, attr) n. Bytes = soc. write(str, len(str)) Grid Programming in Python

Secure MDS Access • Uses the standard Python ldap module with GSI SASL support. import ldap, ldap. sasl l = ldap. initialize("ldap: //scott. lbl. gov: 2135") auth = ldap. sasl. gsigssapi() l. sasl_bind_s("", auth) res = l. search_s("Mds-Vo-name=DSD, o=Grid“, ldap. SCOPE_SUBTREE, "(object. Class=*)") print res l. unbind() Grid Programming in Python

GSISOAP Client Example • Built on SOAP. py, switching to ZSI • GSISOAP client example from py. Globus import GSISOAP, ioc proxy = GSISOAPProxy(“https: //host. lbl. gov: 8081”, namespace=“urn: gtg-Echo”) proxy. channel_mode = ioc. GLOBUS_IO_SECURE_CHANNEL_MODE proxy. delegation_mode = ioc. GLOBUS_IO_SECURE_DELEGATION_MODE_NONE print proxy. echo(“spam, eggs, and spam”) Grid Programming in Python

GSISOAP Server Example from py. Globus import GSISOAP, ioc def echo(s, _SOAPContext): cred = _SOAPContext. delegated_cred # Do something useful with cred here return s server = GSISOAPServer(host. lbl. gov, 8081) server. channel_mode = ioc. GLOBUS_IO_SECURE_CHANNEL_MODE_GSI_WRAP server. delegation_mode = ioc. GLOBUS_IO_SECURE_DELEGATION_MODE_FULL_PROXY server. register. Function(SOAP. Method. Sig(echo, keywords=0, context=1), “urn: gtg-Echo”) server. serve_forever() Grid Programming in Python

py. Globus Requirements • Python 2. 0+ • Globus 2. 0 • Support for dynamic libraries Grid Programming in Python

Future Plans • Windows support • Develop OGSI client modules • Develop an OGSI server framework — Based on Web. Ware • Explore XML based GUI description tools for the automatic generation of GUI components — Should work with both the Java and Python Co. G Kits Grid Programming in Python

Contacts / Acknowledgements • http: //www-itg. lbl. gov/ • krjackson@lbl. gov • This work was supported by the Mathematical, Information, and Computational Science Division subprogram of the Office of Advanced Scientific Computing Research, U. S. Department of Energy, under Contract DE-AC 03 -76 SF 00098 with the University of California. Grid Programming in Python