Cloud Accounting Survey of current status Andrea Guarise

Cloud Accounting Survey of current status Andrea Guarise – guarise@to. infn. it Bologna- Giornata di studio sul Cloud Computing 6 Febbraio 2013

Accounting (is not monitoring) Accounting: The collection of resource consumption data for the purposes of capacity and trend analysis, cost allocation, auditing, and billing. Accounting management requires that resource consumption be measured, rated, assigned, and communicated between appropriate parties. [RFC 2975] …life in the cloud may be not so easy… It’s rather clear hence that the corner-stone of an accounting system is being able to collect the resource consumption data, which is often the most difficult part in implementing an accounting process. Two main classes of resource consumption information (or metrics type) can be identified: Metrics derived from an instantaneous measure of one of the system properties. Metrics derived from an integrated measure of one of the system properties. Accounting is *usually* about keeping track of integrated measures. But…

The. Next. Best. Cloud. Stack

Proposed usage record • • A Cloud accounting usage record is being proposed within the context of the Egi. eu fedcloud taskforce. A mechanism (APEL-SSM) to send those records to a central repository is also already defined. We surely need to comply with this requirement, and as for grid-accounting we also want to store the cloud-accouting records in a national repository… …and, if needed, to extend the usage record to metrics which may not be of interest in the fedcloud context. The first step (to Keep It Simple) will be to identify how to fill *this* usage record with the cloud stacks that we already use. This is the EGI fed. Cloud TF Usage Record version 0. 2

Survey-Open. Stack Native support for accounting within Open. Stack suffers from the difficulty to get usage metrics properly differentiated per user: Open. Stack reports accounting information on a ‘per tenant’ basis where a tenant is basically a project grouping the users. Some metrics are sematically unclear, as an example, start time and end time: do these refer to the start/end of the VM life cycle or to the actual timestamp when the VM start/stops running? In the REST API documentation there’s no reference to network accounting metrics. In general there is not much documantation about accounting.

Survey - Open. Nebula provides a command line tool and a set of Ruby API which allows to retrieve a lot of information that can be used for accounting. Strangely enough the command name is : ”oneacct” It provides three output format: Human Readable, JSON and XML. While the Human readable form is really synthetic and not that useful, both the XML and JSON output are really detailed. The only issue is that not all the measures provided are of the ‘integrated’ type, some of them (e. g the percentage of CPU consumption) are instantaneous, reporting the last available value metered by the underlying monitoring probes, and no corresponding ‘integrated’ values *seems* to be available. These type of fields are however not many and most of the needed fields are available. Some fields seems to be available but need to be further investigated (e. g. suspend. Duration, cpu. Duration and disk)

Survey - WNo. De. S Accounting support in WNo. Des, is yet a work in progress. Not all the relevant fields are covered or there isn’t a full comprehension on how to apply existing information to the record fields. Fortunately enough, WNo. De. S is developed within INFN so it will not be difficult to add the support for the missing fields.

Usage. Record survey Key open. Nebula open. Stack WNo. De. S VMUUID Yes 3 Site Yes Yes Local. VMID Yes Yes clear 2 Local. User. Id Yes Not ? 5 Local. Group. Id Yes Not clear 2 ? 5 Status Yes Yes Start. Time Yes Yes End. Time Yes Yes ? 5 ? 5 Wall. Duration Yes 4 Cpu. Duration ? 1 Yes ? 5 Cpu. Count Yes Yes Network. Type Yes Network. Inbound Yes Network. Outbound Yes Memory Yes Yes ? 5 Yes Image. Id Yes ? 5 Yes Cloud. Type Yes Yes Global. User. Name FQAN Suspend. Duration Disk ? 5 Storage. Record. Id 1. It seems that in this case Open. Nebula mixes accounting and monitoring information, the reported cpu time is infact the latest instantaneous cpu load measurement obtained for the given VM. 2. It is not clear how to determine accounting information for a give nuser. Open. Stack reports accounting information on a ‘per tenant’ basis where a tenant is basically a project grouping the users. 3. Probably yes, to be confirmed or further investigated. 4. Can be roughly derived from Start. Time and End. Time. 5. This aspect needs to be better investigated, mainly from a semantico point of view.

proposed architecture Usage record APEL SSM publisher sensor hlr. Mon Accounting RESTful Web service quota enforcement Billing sensor DB sensor The. Next. Best. Cloud. Stack

Key questions The proposed architecture is general enough to fit many use cases, Neverthless it is necessary to collect accounting use cases to be sure to properly organize the data structures and the records, Some key aspects need to be defined first, as an example: Is it necessary to account just for ended VMs, or we also need to account for usage consumption of the running ones? Probably the second one , but both sensors design, the REST APIs and underlying DB will be heavily impacted by this decision. Which metrics should we give higher priority?

Where do we go from here? use cases, Uses Cases, USE CASES… We need to prototype the sensors, the RESTful service, the APEL-SSM publisher and the hlr. Mon part. Not to be necessarily done from scratch, we can leverage code and experience gained on the grid, But it is important to take the chance to adopt technologies that where simply not available ten years ago and could result in faster and better development. We need to better understand how to fill the fields of the usage record which are not covered yet and what their real meaning is, Are there usage record fields to be added? Are there already available piece of code, architecture, experience to levarage (or to start with)? Storage accounting: this is a big *? * A part from the need to have a basic knowledge of the storage occupation made by the VM images, there’s the need to better understand the use cases and overall scope of accounting WRT storage consumption. Venus-C, an EU-Project aimed at cloud infrastructures, ended in May-2012, is a good source of references and should be carefully inspected.