Following up Brad Anderson & Srikanth’s blog posts here& here, I would like to create a list of blog posts that describe how to deploy usage metering & reporting in System Center 2012 R2 & Windows Azure Pack for IaaS.
Some would ask why should I care about usage & service reporting for IaaS?
There are a couple of reasons why Service Reporting is important for the Cloud:
- Pay as you go & Pay only for what you use.
This supports the cloud business model where metering is one of the fundamentals of Cloud Computing.
- Elastic consumption.
This requires analytics at the macro and micro levels in order to meter at the tenant level as well as drill down to the single instance under a tenant.
To achieve this it requires significant effort and skills to gather, aggregate and summarize the right set of data. Microsoft is providing this capability as part of System Center 2012 R2.
In this series of blog posts I will focus on Usage and Metering for VM Clouds only, as this service seems to be area that most people start with. The process works very similar for other Windows Azure Pack services, once VM Cloud metering is up and running the, other service areas like Web Sites, MSSQL, MYSQL, ServiceBus could be a natural next step in the usage & metering process.
IaaS Service Reporting & Usage Metering
The overall solution contains the following:
Collecting the data:
- Collecting IaaS usage (CPU, Memory, Storage, Network).
- IaaS data comes from the Operations Manager DW through Service Provider Foundation (SPF).
- Windows Azure Pack provides consistent usage collection for every service offered all service providers such as Web Site, Virtual Machine, Service Bus, SQL and MYSQL servers.
Analyzing and Reporting on the usage data:
- Service Reporting analysis and Report on the collected data by using already existing Microsoft BI stack (SQL Server, Excel Power Pivot & SharePoint Performance Point).
Component overview and data flow.
In System Center 2012 R2 usage metering & reporting for IaaS (VM Cloud Service in Windows Azure Pack) involves the following list of components:
- Virtual Machine Manager 2012 R2 (VMM).
- Operations Manager 2012 R2 (OM).
- Service Provider Foundation (part of Orchestrator installation media) 2012 R2 (SPF).
- Windows Azure Pack (WAP).
- Service Reporting (part of Orchestrator installation media) (SR).
- SQL Server (2008 or 2012).
To dive a little deeper into the process flow, already described by Srikanth (here), you can divide it into the following three high level steps:
- Data collected by System Center (VMM + OM).
- Data collected by Windows Azure Pack (SPF + WAP).
- Data analyzed by Service Reporting (New component in System Center R2).
The illustration above shows how data flows from the different components in the process of usage, metering & reporting.
The following sections will explain each step in the process:
Data collected by System Center
System Center has valuable information about VM clouds. This data is collected by Virtual Machine Manager, in order to get this data aggregated and summarized for later usage by Windows Azure Pack the following needs to happen:
Process Flow
1. Virtual Machine Manager collects data for all VM and Hosts managed by Virtual Machine Manager. This includes metrics like CPU, Memory, Storage, Network, Started and Stopped. This data is only stored for one month by default in the VMM Database.
2. Virtual Machine Manager sends its performance data to Operations Manager by using the VMM to OM connector.
3. Operations Manager saves the raw data in the Operations Manager Data Warehouse.
4. Every hour Operations Manager Data Warehouse moves the raw data to the hourly dataset where it’s stored for one year by default. All the raw data is aggregated into min, max, avg & std deviation for each performance metric.
This process is described in the following picture:
Data collected by Windows Azure Pack (SPF + WAP)
Service Provider Foundation (SPF)
Service Provider Foundation exposes an extensible OData web service that interacts with Virtual Machine Manager (VMM) & Operations Manager (OM). This enables service providers and hosters to design and implement multi-tenant self-service & usage that integrate IaaS capabilities available in System Center 2012 R2. Service Provider Foundation provides usage metering data to any client, such as Service Management Portal and API, that wants to collect it. This data comprises all the usage metering metrics for all the virtual machines that a tenant uses, provided that those virtual machines are being monitored by System Center 2012 – Operations Manager and that the data is being stored in Operations Manager Data Warehouses. The role of Service Provider Foundation is to collect metrics from all the data warehouses and aggregate them for billing and analysis purposes.
WAP Usage Service
Inside Windows Azure Pack, the Usage Service consists of three components:
- Usage Collector
- Usage (REST API)
- Usage Database
Usage Collector
Usage collector consists of a scheduler that triggers the collection of information from the different resource providers in a circular fashion
Each resource provider has a list of data sets that the usage collector will collect.
The time span of the collection cycle is configurable, by default the usage collector spends a max of 3 minutes by default on every resource provider. So every collection cycle is 3 min * Number of Resource Providers in the System. I.e. (Resource Providers: VM, SQL Server, MySQL, WEB) * 3 minutes = 9 mins maximum for a cycle
The Usage system interacts with each resource provider one by one, if the provider returns with an empty response it moves to the next provider in the cycle.
Usage Database
Data from the usage collector stored in the central data repository. The data stored comes from the different resource providers and also contains relationships to tenants and plan.
Data is stored for 40 days by default
Usage
The Usage REST API exposes a REST endpoint for data access. This is used to allow access to the content in the usage database by billing providers and analytics services
The process is as follows for VM Clouds (IaaS) Resources:
1. Usage Collector (a sub component of Usage Service) collect usage data from every Resource Provider.
2. The Usage Collector for VM Cloud Resource collects needed data like CPU, memory and disk.
3. The Usage Collector interacts with the Service Provider Foundation (SPF) metering endpoint (Usage) to collect data for VM Clouds.
4. Service Provider Foundation usage metering module calls Operations Manager Data Warehouse and returns the data to the Usage Collector service.
5. The Usage Collector stores the data in the Usage Database.
Orange: Internal Modules
Yellow: REST endpoint
Blue: WAP / SPF Component
Green: WM Clouds Resource Providers and sub-components (SPF & System Center)
Service Management & Usage Data Model
The usage data model is used to store the usage data for the tenants consuming the Resource Provider services collected through Usage Collector.
This enables mapping between a Resource provider instance (e.g. VM) usage to be associated with a tenant (user) and the subscription that a given tenant signed up for.
For Service Management the following concepts are used:
Tenant: One or more individuals consuming services from the Windows Azure Pack via a subscription.
Usage Record: A data record that indicates the resource consumption per Tenant, per subscription and per provider.
Subscription: The mapping between a tenant and one or more plans.
Plan: Contains one or more services (Resource Providers: VM, DB, Web, Service Bus) offered and the quota for each service. Subscriptions, Add-on and Resource Providers are all associated to a plan.
Add-on (Plan Add-on): Allows customer upgrades on quota or add capabilities that can be combined with a plan (a service can be optional) to extend functionality or quota within a plan as an extra service.
Resource Providers: Services offered within Windows Azure Pack with mapping to resources (e.g. Web Sites or Virtual Machines).
The Usage Service collects data from Service Management module by using REST API and stores this data in the Usage Database
The usage database now contains both usage data from the different resource providers and the relation ships between tenants, plans and resource providers.
Data analyzed by Service Reporting
A new component introduced in System Center Orchestrator R2 is Service Reporting. This component does macro and microanalysis based on data collected from various resource providers in the Windows Azure Pack. (VM, DB, Service Bus & Web).
1. Service Reporting uses ETL (Extract, Transfer Load) standard to collect data (This is much similar to the way Service Manager Data Warehouse collect data from Service Manager DB).
2. The Extract process will contact the Windows Azure Pack Usage API to extract data.
3. WAP Usage API will return the data from the usage database to the extract process.
4. After completing the ETL process, the data is transferred and stored in Cubes for analytics purpose.
5. After this has completed, the data can be analyzed by Excel (PowerPivot) or SharePoint (PerformancePoint).
Below are two examples for usage data analyzed in Excel (PowerPivot) & SharePoint (PerformancePoint)
To summarize the flow the following diagram shows the process end to end:
I hope that this blog post provides a basic understanding of how Usage Metering & Reporting for IaaS works in System Center 2012 R2 and the Windows Azure Pack.
In upcoming blog posts, I will describe how each component described above can be configured to make use of IaaS usage and metering.
- Configuring VMM and OM for IaaS usage and metering.
- Configuring SPF and Windows Azure Pack for IaaS usage and metering.
- Installing & configuring Service Reporting for IaaS usage and metering.
I would like to thank the Windows Azure Pack and Reporting Services product groups for providing inputs and reviewing this blog post.
As always, I am interested in feedback and suggestions.