SZTAKI International Desktop Grid Federation status report Jzsef

SZTAKI & International Desktop Grid Federation: status report József Kovács (MTA SZTAKI) jozsef. kovacs@sztaki. mta. hu http: //idgf-sp. eu IDGF-SP is supported by the FP 7 Capacities Programme under contract nr RI-312297 1

International Desktop Grid Federation: short introduction Legal form: Dutch foundation Main vehicle for dissemination during the lifetime of the EC funded projects and for continuation after the projects Federation is open for organisations and individuals in research and industry A member organisation with chapters and working groups Focal point for DCI/Desktop operators and application developers Webportal: discussion fora, events and workshops, support tools, etc. Provide assistance and facilities to launch new joint actions, initiatives, projects 2

IDGF-SP International Desktop Grid Federation - Support Project Supported by: EC in the FP 7 Project acronym: IDGF-SP Contract n°: RI-312297 Project type: CSA-SA Start date: 01/11/2012 Duration: 24 months Funding from the EC: 860. 000 € Total funded effort in person-month: 136 Web site: http: //idgf-sp. eu Coordinator: Dr. Robert Lovas email: rlovas@sztaki. hu Involve and engage in long-term significantly more citizens and new communities in the • volunteer and • private (campus-wide or enterprise) Distributed Computing Infrastructures by supporting the (1) rapid creation, (2) efficient operation, and (3) dynamic expansion of this type of DCIs for e. Science. Coordinate and synchronise the dissemination and support activities of major European stakeholders of volunteer and Desktop Grids with focus on the International Desktop Grid Federation. 3

SUMMARY: software components of maintained by SZTAKI • • • SZDG boinc package: debian variant of BOINC supported by our tools 3 G Bridge: remote job submission, converting incoming jobs to WUs DC-API: c library for creating master-worker applications Genwrapper: porting legacy application to BOINC, shell interpreter GBAC: virtualisation under BOINC client, based on virtualbox Metajob: submission of large number of jobs to 3 Gbridge in one turn Cloud handling by 3 G Bridge: dynamic on-demand cloud bursting Monitoring by 3 G Bridge: information collection on job throughput EGI: g. Lite CREAM m. CE: automatic job transfer from g. Lite to BOINC EGI: Munin and Nagios: EGI compliant monitoring g. USE: web-based graphical portal for executing jobs and workflows 4

Documentation site: doc. desktopgrid. hu 5

Recent (ongoing) activities and plans (I. ) • Supporting the latest BOINC release by our tools (DC-API, 3 GBridge, …) • plan is to put DC-API into the BOINC trunk • testing has been successfully done • need David to integrate our staff into the BOINC code repo • Improving GBAC • handling of stdout & stderr of application has been improved • returning the log of the internal GBAC component for debugging purposes • automatic image extraction on the client side (image size 700 MB -> 100 MB) • short term plan is to support image selection (discussing what is the easiest/most efficient way to implement it…) • Improving 3 GBridge • 3 GBridge supports REST interface, implementation finished • http: //doc. desktopgrid. hu/doku. php? id=manual: restif • consumed CPU hours, credits can now be queried (required for EGI accounting & ELTE) 6

Recent (ongoing) activities and plans (II. ) • Volunteer Storage • Using random linear coding in distributed storage • See presentation from Adam Visegradi • New applications on our public, volunteer projects • BIOVEL community applications on EDGe. S@home • Biome-BGC ecosystem modeling is the estimation of behavior of terrestrial ecological systems (forests, grasslands, agricultural areas) as accurately as possible. • New mathematical application will appear on SZTAKI Desktop Grid these days • Riemann zeta research project: http: //www. riemann-siegel. com/ • Locating large values of zeta functions on the critical lines • New user community: DIRAC • The DIRAC system has been recently integrated with BOINC through GBAC • The GBAC VM image has been adjusted to the needs of DIRAC • DIRAC jobs are currently arriving through g. Lite • Testing finished, deployment on EDGe. S@home will happen these days 7

Recent (ongoing) activities and plans (III. ) • Cloud scenarios • We created cloud images to support automatic setup of BOINC server with clients • To ease the deployment • To perform dynamic, on-demand computing infrastructure for scientific users • To encourage scientific users to utilise BOINC • We are creating images for each scenario defined at doc. desktopgrid. hu • Image with BOINC client • Needs a proper configuration to force the client to continuously execute WUs in the cloud • Not so easy in the new scheduling algorithm, recompilation is needed • Adjusting related parameters hardwired in the client_state. h would help… • #define WORK_FETCH_PERIOD 60 • #define WF_MIN_BACKOFF_INTERVAL 600 • #define WF_MAX_BACKOFF_INTERVAL 86400 • #define WF_DEFER_INTERVAL 300 8

European Grid Infrastructure (short overview) • European • Over 35 countries • Grid • Secure sharing of IT resources • Infrastructure Coordinator: EGI. eu, Amsterdam • Computers (clusters): 372, 612 CPUs • Data: 180 PB disk • Applications: 1. 67 million job/day (thousands of scientists) • …. and beyond!! • Technologies to • • Federate resources Access resources 9 9

Proposed EGI DG-VT Project • General project info • Start (planned) : September-October 2013 • Duration : 6 months • Leader: MTA SZTAKI from the Hungarian NGI • “EGI-In. SPIRE is ideally placed to join together the new Distributed Computing Infrastructures (DCIs) such as clouds, supercomputing networks and desktop grids, for the benefit of user communities within the European Research Area. ” --- EGI-In. SPIRE Project fact sheet • Motivation • Despite the on-going work and the benefits of the integrated Desktop Grid DCI resources available for EGI users, only a part of the EGI community is aware of the latest Desktop Grid related achievements, and a fraction of the EGI users and infrastructure operators take their advantages in everyday practice. • In order to fill this gap; a part of the targeted objectives are technical ones but there is stronger focus on the human aspects – training, networking and support activities. • See list of Virtual Teams at https: //wiki. egi. eu/wiki/Overview_of_Virtual_Team_projects 10 10

Proposed EGI DG-VT Project • Promote and train the Desktop Grid related technologies in the EGI communities • Utilize the available bridged Desktop Grid resources by more VOs • Increase the number of heavily used EGI applications on the integrated (Desktop Grid) infrastructure with focus on widespread tools/solutions/approaches • Improve documentation, e. g. Road maps, training materials, manuals • Complete the final remaining steps for full integration concerning e. g. dashboard • Find joint EGI champion(s) - IDGF ambassador(s) 11 11

Current status of integration into the EGI infrastructure https: //wiki. egi. eu/wiki/Desktop_Grids_integration 12

EGI related activities and plans • EGI activated its efforts in full integration of the Desktop Grid systems/projects to the EGI infrastructure and in promoting Desktop Grid among the EGI scientific users • EGI plans to put manpower behind as well • Plans: • EDGe. S@home will be renamed to EGI@home • Technical discussion is needed, how to do that? Share your experiences if any… • • • Screensavers for the EGI applications Use the cooperation with BIOVEL and DIRAC as a positive example More dissemination (newsletters, leaflets) to notify scientific users about DG resources 13

Thank you for your attention… Presenter: Jozsef Kovacs (jozsef. kovacs@sztaki. mta. hu) Project website: http: //idgf-sp. eu http: //desktopgridfederation. eu http: //www. lpds. sztaki. hu Acknowledgements: Ø The presented work is supported by the FP 7 Capacities Programme under contract nr RI-312297 (IDGF-SP). 14