RGMA Producers and Archiver Paul Mealor Overview gsiwuftp

R-GMA Producers and Archiver Paul Mealor

Overview gsiwuftp edg-netmon-archiver edg-gridftplog 2 rgma Log P re He Producer API Archiver API be My. SQL DB P ts le rv Producer API P se edg-netmon-rgma-info A C edg-pinger-timeping Producer API FIFO edg-netmon 2 rgma DBP

edg-netmon 2 rgma • Maintains producers for the tools – Tools’ scripts are rerun for each measurement batch – Program running APIs must be persistent • Files – /opt/edg/sbin/edg-netmon 2 rgma. pl – /opt/edg/var/edg-netmon 2 rgma-fifo – /etc/rc. d/init. d/edg-netmon 2 rgmad • Configuration – RGMA client configuration

Tool/netmon 2 rgma interaction Ping. ER Normal log Web interface • addmetric pinger rtt 10 NMId. Source=pc 18. hep. ucl. ac. uk tool=pinger edg-netmon 2 rgma If producer doesn’t exists If buffer. Size < count * 4 / 3 • Circular. Buffer. Producer_new(“Network. RTT”, “WHERE NMId. Source=‘pc 18. hep. ucl. ac. uk’ AND tool=‘pinger’”); • Circular. Buffer. Producer_set. Remote. Buffer. Size($producer, 10*2);

Tool/netmon 2 rgma interaction Ping. ER Normal log Web interface • measurement pinger rtt pc 18. hep. ucl. ac. uk edgnm. gridpp. rl. ac. uk pinger 100 57 edg-netmon 2 rgma • Circular. Buffer. Producer_insert($producer, ”INSERT INTO Network. RTT (NMId. Source, NMId. Destination, tool, packet. Size, value) VALUES (pc 18. hep. ucl. ac. uk, edgnm. gridpp. rl. ac. uk, pinger, 100, 57)”);

edg-netmon 2 rgma • Possibly should rewrite in Java – Problem in C (Perl) API results in memory leak and heavy performance hit – Current solution is to restart the daemon with a cron job • /etc/cron. d/edg-netmon 2 rgma-restart – But java lacks the text-handling of Perl – Would have to have java on the NM

Tools modifications • Generally 2 new functions • initialise_rgma – Writes initialisation directives to the FIFO – i. e. fixed columns, metric names, expected number of entries • Netmon 2 rgma creates producers, changes buffer sizes when necessary • produce_it – Writes the results of the measurements • No extra configuration – Assumes /opt/edg/var/edg-netmon 2 rgma/edgnetmon 2 rgma-fifo

edg-netmon-rgma-info • Publishes Network. SE, Network. CE tables – Republishes every 12 hours (i. e. keepalive) • Files – /opt/edg/sbin/edg-netmon-rgma-info. pl – /opt/edg/etc/edg-netmon-rgma-info. conf • Local NM, associated SEs and CEs – /etc/rc. d/init. d/edg-netmon-rgma-infod • RGMA client configuration

edg-ftlog 2 rgma • Publishes file-transfer logs (i. e. Grid. FTP) • Files – /opt/edg/sbin/edg-gridftplog 2 rgma. pl – /etc/rc. d/init. d/edg-gridftplog 2 rgmad • Reads from Grid. FTP log – /var/log/gsiwuftpd. log – Publishes new entries when they are added • RGMA client configuration

edg-netmon-archiver • Program – /opt/edg/share/edg-netmon-archiver. jar – /opt/edg/sbin/edg-netmon-archiver • runs the Java code – /opt/edg/bin/edg-netmon-archiver-setup • Configures My. SQL database; requires mysql root password • What to do in LCFG? – /opt/edg/etc/edg-netmon-archiver. conf • List of tables to archiver plus predicates for each – /etc/rc. d/init. d/edg-netmon-archiverd – Also RGMA client configuration

edg-netmon-archiver • Ironically this might be the best package to write in C or Perl – Archiver API is all but unaffected by C API problem – But might run on R-GMA servlet machine • Interaction with edg-netmon-info – Archives every row that edg-netmon-info produces – Perhaps need to automatically remove duplicates (via mysql)

Package progress • Tools, edg-netmon 2 rgma, edg-netmon-rgma-info – 99% – Latest tweaks still need to be uploaded • edg-ftlog 2 rgma – 75% – rc script, autobuild process • edg-netmon-archiver – 95% – need to upload, tag, check build process (ant)

Testing • • Tools on pc 18. hep. ucl. ac. uk edg-netmon 2 rgmad on pc 18. hep. ucl. ac. uk “User servlets” on pc 23. hep. ucl. ac. uk Registry and Schema servlets on pc 23. hep. ucl. ac. uk – Something always goes wrong when I try to connect to the WP 3 Registry at infocat. gridpp. rl. ac. uk • >1200 RTT/loss measurements archived by Tuesday • >130 TCP throughput measurements to localhost : )

Grid. FTP logging in MDS Paul Mealor

Status • Information provider – Will be in EDG 2 on SE • Publishes the most recent information for each host pair • MDS being phased out in EDG 2 – GIIS structure replaced by R-GMA – Will be all R-GMA later

Details • Publishes achieved throughput – Filesize, time etc as parameters • Publishes most recent transfer between each host pair • Specifies source and host pair, plus, if known, source and host NM – NM names taken from config file as with other netmon info providers • Uses same schema as other netmon – But doesn’t use the full tree structure

Schema dn: NMMeasure. Id=$source/$dest, $opt{rootdn} object. Class: Network. Measurement NMMeasure. Id: $source/$dest Source. Host: $source Source. NE: $source. NE Dest. Host: $dest Dest. NE: $dest. NE Metric. Name: gridftpthroughput Metric. Value: $throughput Metric. Unit: bit/s Monitor. Tool: Grid. FTP Metric. Time: $timestamp Parameter: time: $etime Parameter: size: $size Parameter: buffersize: $buffer Parameter: streams: $streams Parameter: stripes: $stripes create. Timestamp: $timestamp modify. Timestamp: $timestamp

END