Grid HTTPHTTPS extensions 18 November 2002 Andrew Mc

Grid HTTP/HTTPS extensions 18 November 2002 Andrew Mc. Nab, University of Manchester mcnab@hep. man. ac. uk Andrew Mc. Nab - Grid HTTP/HTTPS extensions

Overview u HTTPS u HTTP as a grid protocol as a data protcol u Multistream u Grid HTTP: curl-get HTTP/HTTPS usage u G-HTTPS u Trusted Caches u file. Grid. Site HTTPS server u Third Party Transfers u curlfs for Slash. Grid u Summary Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

HTTPS as a Grid protocol u HTTPS is an interesting and important protocol for several reasons: n it is by far the most widely deployed secure protocol n has a large amount of high quality software that we could leverage n has excellent interaction with Firewalls, Network Address Translation and Application Proxies s n has the potential to solve some of the problems sites have with private IP farms along with HTTP, is the basis for Web and Grid Services u HTTPS n consists of HTTP/1. 1 over an SSL connection security done by SSL layer, using X 509 certificates (including GSI) u HTTP/1. 1 (rfc 2616) and extensions like Web. DAV (rfc 2518) have a rich set of methods (GET, PUT, DELETE, COPY etc) headers (“Expires: ” etc) and Errors (“ 413 Request Entity Too Large”) n so a standard way exists for many of the transfer operations we need Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

HTTP as a data protocol u Same advantages as HTTPS: large amount of existing high quality software, and good operation with Firewalls, NAT etc. u If we build secure HTTPS information/control services, easy to provide HTTP data services: n n Do GET during HTTPS session, but server responds with redirect to HTTP data server? So Grid. FTP Control & Data channels --> HTTPS Negotiate and HTTP Data connections u Kernel-based n need to do something like that to fully use a machine’s gigabit interface u HTTP n “zero-copy” HTTP servers like tux are very efficient connection and a Grid. FTP data channel are same at TCP layer but may want a way to specify TCP parameters to be used by HTTP server responding with data Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

Multistream HTTP u HTTP can support application-level multiple streams and striping by using the standard Range: header from RFC 2616 (HTTP/1. 1) to set up many partial fetches. u This n mechanism is supported by almost all modern web servers eg Apache and Red. Hat’s tux kernel httpd u Multiple n n streams implemented by client splitting into threads Each thread requests a block of the file from the server As each request completes, thread finds next unfetched block and requests it u Striping by doing the same mechanism, but with more than one server u curl-get n demonstrates both of these source is 300 lines of C, in EDG CVS Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

curl-get examples u Rough n tests done, copying files from Manchester to CERN elapsed times in seconds, average of 10 copies of each type, alternated Size 292 M 29 M 2. 9 K curl-get 64. 6± 6. 1 96. 0± 9. 2 7. 1± 1. 4 31. 6± 0. 49± 0. 07 3. 30± 0. 16 0. 11± 0. 00 globus-url-copy 62. 1± 4. 9 74. 8± 3. 8 6. 9± 1. 8 15. 9± 0. 9 2. 24± 0. 10 2. 61± 0. 18 2. 15± 0. 04 1. 05± 0. 10 streams 20 5 20 1 Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

Extensions to HTTPS/HTTP u HTTPS/HTTP already have most of the functionality we need for Grid information/control/data transport n n some of these come from several sources (eg the Web. DAV RFC 2518 not just HTTP/1. 1 itself) and can be done different ways so want to specify a sufficient subset for interoperability u However, can identify some extensions that are also needed: n delegation to HTTPS n some way of returning access control information along with data n other metadata too n may want to specify TCP parameters for bulk data tranfer Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

“G-HTTPS” u. A proposal by Akos and me, for backwards compatible extensions to HTTPS n discussed on wp 2 -sec and wp 7 -security lists u Adds GSI proxy delegation to HTTPS using additional methods (eg PUT-PROXY) and headers (eg Delegation-ID) u Allows n n services to return generalised metadata in headers or by URL initially this allows services to return the GACL of a response for more efficient caching (ie sharing cached copies with other users. ) essential to include expiration and caching policy information too u Aim is to avoid breaking existing HTTPS systems and to achieve “pass through” compatibility: n even if HTTPS client or server software doesn’t understand extensions, they can make them available to the application which does Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

Example of delegation by HTTPS u Client issues GET-PROXY-REQ request, perhaps with a message body specifying any extensions required in the proxy cert u Server generates a key and a certificate request, returns this in the response message body. u Client signs this, and returns it in the body of a PUT-PROXY request u Need a Delegation-ID header in the above exchanges so can keep track of the delegation session n may want to maintain delegation sessions for the same user at one server, but with different amounts of delegation u Subsequent GET, PUT etc actions carry on using the Delegation-ID u Non G-HTTPS server will respond with “ 501 Method not implemented” to above methods Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

Application of delegation: Trusted Caches u Many information services are going to need delegation, but Trusted Caches are one purely file transfer application of this u Existing HTTPS isn’t cache-able: n connection from client to origin server for trust to mechanism work n So best you get is opaque proxying/tunneling of SSL u With delegation, can improve this: n identifies a caching server it trusts (in its VO maybe? ) n delegates a credential to it n makes an HTTP proxy request via HTTPS: GET http: //a. b. c/def n caching server fetches this using delegated credential, gives it to client n n if can get an ACL for this file, may be able to return file from cache in subsequent requests also means that only real HTTPS works, not other things hidden in SSL Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

file. Grid. Site u Read (GET) well supported by HTTPS servers. u However, write (PUT, DELETE, MOVE, COPY) usually left to CGI programs, servlets etc. n Access control also usually limited to client IP or HTTP passwords. u file. Grid. Site Apache adds Grid authorisation and write operation support to n a cut-down version of Grid. Site (used for https: //marianne. in 2 p 3. fr) n file rather than webpage orientated (no fancy headers on HTML etc) n uses GACL to handle the Access Control Lists n can work with mod_ssl-GSI so clients can authenticate with a GSI proxy u Turns an Apache webserver into a Grid HTTPS fileserver with the key functionality of a Grid. FTP server. Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

file. Grid. Site examples with curl u Curl is a standard HTTP/HTTPS command line client (cf wget) u Get a file using GSI proxy in /tmp/x 509 up_u 100 n curl --capath /etc/grid-security/certificates/ --cert /tmp/x 509 up_u 100 https: //a. b. com/example 1. txt u Copy n a file to the file. Grid. Site server with HTTP PUT: curl --capath /etc/grid-security/certificates/ --cert /tmp/x 509 up_u 100 --upload-file /tmp/example 2. txt https: //a. b. com/example 2. txt u Delete n curl --capath /etc/grid-security/certificates/ --cert /tmp/x 509 up_u 100 --request DELETE https: //a. b. com/example 2. txt u Create n a file with HTTP DELETE: a directory with PUT to …/ curl --capath /etc/grid-security/certificates/ --cert /tmp/x 509 up_u 100 --request PUT https: //a. b. com/newdir/ Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

Adding delegation to file. Grid. Site u Doing this as a demonstration of G-HTTPS extensions u Delegation n n needed for Third Party Transfers Use COPY from Web. DAV RFC 2518 which allows source or destination to be absolute URL’s Spec actually allows “fourth party” too, involving two remote URL’s and the transfer being tunneled through the server. u Delegation also useful for fileservers which need credentials to access local storage n to get token for local AFS cell (Lyon have had to work around this with Grid. FTP servers) Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

curlfs for Slash. Grid u curl n is built on top of a general library, libcurl handles persistent HTTP and HTTPS connections, SSL setup etc u To add HTTP and HTTPS filesystems to Slash. Grid, have made a libcurl filesystem plugin: curlfs u This n maps parts of the URL space into the local filesystem: https: //a. b. com/newdir/ ---> /grid/https/a. b. com/newdir/ u Works n with any standard HTTP or HTTPS server rpm -i /grid/http/datagrid. in 2 p 3. fr/distribution/globus/beta-21/RPMS/* u Slash. Grid framework provides GSI proxy or full cert/key to curlfs so it can make authenticated requests. u Write n with HTTP/1. 1 PUT and DELETE being added to curlfs Will complement file. Grid. Site support for these on server side Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002

Summary u HTTPS n G-HTTPS extensions being worked out u HTTP n as a grid protocol as a data protocol even a quick multistream HTTP hack seems very competitive u file. Grid. Site HTTP(S) server has been written n supports read/write with standard utilities like curl n third party transfers being added as demonstration of delegation u curlfs written for Slash. Grid: maps URL’s into filesystem u Source u See code for curl-get, file. Grid. Site, curlfs is in EDG CVS http: //www. gridpp. ac. uk/authz/ for more details Andrew Mc. Nab - Grid HTTP/HTTPS extensions - 18 Nov 2002