Enabling Grids for Escienc E Grid Accounting tool
Enabling Grids for E-scienc. E Grid Accounting tool: DGAS Giuseppe Patania INFN Torino www. eu-egee. org INFN tutorial – June 4 -5 Bologna 1
Enabling Grids for E-scienc. E Summary • Overview • Architecture • Components • Features • Security & Privacy • Deployment • Client tools www. eu-egee. org INFN tutorial – June 4 -5 Bologna 2
Enabling Grids for E-scienc. E Accounting overview – In a distributed computing environment people want to know who used the resources and how many resource have been used. – Users : How many resources am I using ? – VO managers : how many resources is my VO using ? – Site Managers : who is using my resources ? – A good accounting system should be able to answer these questions taking care of all the security and privacy issues. www. eu-egee. org INFN tutorial – June 4 -5 Bologna 3
Enabling Grids for E-scienc. E DGAS architecture • DGAS is a distributed system able to perform a resource usage metering and Economic Accounting (as a basis for billing) in the Grid environment. • It is based on a client/server infrastructure relying on a network of independent servers. www. eu-egee. org INFN tutorial – June 4 -5 Bologna 4
Enabling Grids for E-scienc. E DGAS Components • Sensors on CEs (urcollector and pushd) – Urcollector: collect usage records into a file, from LRMS and gatekeeper logs – Pushd: sends usage records to HLR using the atm. Client() function. • Resource (site) HLRs (Multilevel structure) – HLR 1 L: Stores job’s data, received from registered sites, into a My. SQL db and forwards all to the enable HLR 2 L – HLR 2 L: it is a data aggregator, receives data from several HLR 1 L www. eu-egee. org INFN tutorial – June 4 -5 Bologna 5
www. eu-egee. org Enabling Grids for E-scienc. E DGAS Components II • User HLRs – User HLR: logically different from reosurce HLR. It stores data for users, every registered user can do queries to retrieve only his own data. – Not installed on the italian grid nodes. • Local and remote query client tools – Allow a user, a HLR admin, a VO admin retrieving data info – Every HLR server has a local tool set of clients for querying database. – In every user interface a specific remote clients tool set can be installed. INFN tutorial – June 4 -5 Bologna 6
Enabling Grids for E-scienc. E DGAS Workflow Usage Record L 2 VO HLR 3 3 Usage Record 3 Site HLR User HLR CE 1 job 2 job WN www. eu-egee. org INFN tutorial – June 4 -5 Bologna 7
Enabling Grids for E-scienc. E Usage records • Different types of URS • Usage quantities: – cpu time, wall clock time, phisical memory used virtual memory used, spec benchmark units • Timestamps: – job execution start and end, job creation on LRMS, job insertion in the last LRMS queue, etc. www. eu-egee. org INFN tutorial – June 4 -5 Bologna 8
Enabling Grids for E-scienc. E Usage records II • User related: – Local User id on the CE – User VO – Local User Group on the CE – User voms FQAN • Job related – Job grid id – Local LRMS job id on the CE www. eu-egee. org INFN tutorial – June 4 -5 Bologna 9
Enabling Grids for E-scienc. E Usage records III • Host related: – Resource HLR destination address – Default user HLR address, if available – User VO – Resource Grid Id • Other – SITE NAME , etc www. eu-egee. org INFN tutorial – June 4 -5 Bologna 10
Enabling Grids for E-scienc. E DGAS Features • Granularity – Resource accounting at single job level or in aggregate form per user, per VO, per resource (site) • Scalability – Arbitrary number of Resource/VO HLRs can be deployed. • Hierarchical Design – HLRs can be interconnected across a tree structure, so multiple levels of aggregation are possible. www. eu-egee. org INFN tutorial – June 4 -5 Bologna 11
Enabling Grids for E-scienc. E DGAS Features II • Completeness – Capability to collect information both for grid and local jobs. • Designed to perform pricing ( but not used) – Possibility to assign “computing credits” (something similar to quotas) that can be used as a basis for billing the resources consumption. www. eu-egee. org INFN tutorial – June 4 -5 Bologna 12
Enabling Grids for E-scienc. E Security and Privacy I • Information confidentiality is guaranteed by the use of different authorization levels to access the Usage Records. – Users (can access their own detailed records and aggregates) – Site Managers (Can access their own site detailed records and aggregates) – VO Managers (Can access detailed records and aggregates of all VO members) www. eu-egee. org INFN tutorial – June 4 -5 Bologna 13
Enabling Grids for E-scienc. E Security and Privacy II • Full VOMS integration in query authorization is available (now on L 2 HLR, on every HLR in future releases) – (e. g. /atlas/Role=vomanager/Group=NULL) • Integrity of the data flow is guaranteed by the use of GSI and data encryption. • No private information is sent in clear text www. eu-egee. org INFN tutorial – June 4 -5 Bologna 14
Enabling Grids for E-scienc. E Deployment Italian Grid (L 2 HLR) DGAS concentrator Torino HLR Barii HLR Milano HLR Padova INFN-ROMA 1 -2 -3 . CE Padova CE Trieste CE Ferrara CE Torino CE Bari CE Milano CE Bologna CE Perugia CE Firenze CE Genova www. eu-egee. org INFN tutorial – June 4 -5 Bologna 15
Enabling Grids for E-scienc. E Client tools • Remote clients for querying HLR are provided • Actually, because none HLR users are installed, • remote clients are allowed only for two kind of people: – Hlr admins – VO admins • Every one wich belongs to one of the two categories above can exploit the glite-dgas-hle-query tool to interact with both HLR 2 L for data retrieving • Every admin must be mapped into the hlr server he wants to connect to. www. eu-egee. org INFN tutorial – June 4 -5 Bologna 16
Enabling Grids for E-scienc. E Checking the server status It is possible to use the ping command to check the server status and the engines available www. eu-egee. org INFN tutorial – June 4 -5 Bologna 17
Enabling Grids for E-scienc. E Querying the HLR 1 L Here two queries are shown, aggregated for resource and for VOS www. eu-egee. org INFN tutorial – June 4 -5 Bologna 18
Enabling Grids for E-scienc. E Querying the HLR 2 L • In the following, an example of query to the HLR 2 L using the glite-dgas-hlr-query command: www. eu-egee. org INFN tutorial – June 4 -5 Bologna 19
Enabling Grids for E-scienc. E Checking admins on HLR 2 L Checking admins, and their roles www. eu-egee. org INFN tutorial – June 4 -5 Bologna 20
Enabling Grids for E-scienc. E …. . questions ? www. eu-egee. org INFN tutorial – June 4 -5 Bologna 21
- Slides: 21