Auditing the Microsoft Private Cloud with ACS Autopilot - Part 3

Implement ACS Autopilot: Part 3 of 3 posts in a “mini-series” describing how to automatically audit private cloud enterprises.

As stated in part 1 of this series, the task of enabling ACS forwarders on target endpoints is normally a manual procedure–and for good reason! If the automation in the accompanying “autopilot” management pack were to be enabled prematurely or without careful planning, it could potentially overwhelm the ACS backend infrastructure. However, much like the “autopilot” technology can be a good thing for flying planes and has a role to fulfill in the course of a flight, the automation specified in this management pack should only be implemented under certain conditions. “Kids, don’t try this at home.”

As reflected in the graphic above, the process of enabling ACS forwarders in a safe manner is manual at first. Then, once ~99% of the forwarders been enabled, turn on ACS autopilot to audit all target computers in a dynamic and elastic manner.

Assumptions

The following activities are assumed to have already been followed and successfully implemented in the target environment:

• Audit requirements fully factored into the design of ACS
• Appropriate level of audit policies implemented and applied against target computers to meet governance requirements
• ACS infrastructure, including storage configuration, adequately sized to address anticipated audit volume
• ACS filter properly tested and verified against audit requirements and successfully implemented on the ACS collector(s)
• Operations Manager agents automatically deployed to target computers by Configuration Manager software distribution package
• Target computers are limited to Windows servers

Note: This management pack is compatible with both OpsMgr 2007 R2 and OpsMgr 2012 infrastructures.

Implementation

Import the accompanying management pack (Microsoft.SystemCenter.ACS.Extended.xml) into the Operations Manager 2007/2012 management group and then follow these steps over the duration of the implementation of ACS forwarders:

1. In the Operations Console, click the Monitoring wunderbar. In the Monitoring navigation pane, expand the ACS folder and select the ACS Agents – Not Running view. After several minutes, agents with the AdtAgent service not running will gradually populate this view.
2. Manually enable ACS forwarders with one hand on “the switch” (task) and one hand on “the dial” (perfmon):

• Select 10 ACS forwarders in the view by holding the CTRL button on the keyboard down and multi-selecting the target computers’ Health Service objects. Select Enable Audit Collection task in the Actions pane.
• In the Enable Audit Collection task dialog box, click the Override button. Enter the ACS collector(s) FQDN in the provided field; for example: ACS1.contoso.com, ACS2.contoso.com. Click the Override button to commit the change. Click the Run button to execute the task against the target computers.
• Using Performance Monitor on the collectors (or the Performance views in the Microsoft Audit Collection Services management pack in the Operations Console), monitor both the ACS Collector/Connected Clients and ACS Collector/Incoming Events/sec counters. Observe Database Queue % Full counter and throughout the surge and make note of the events per second (eps) that the system can withstand. Wait until the surge in new events levels out before continuing.
• Repeat this iterative step on another 10 ACS forwarders while monitoring the ACS Collector performance counters. If running Operations Manager 2012, increase the number to 20 forwarders and observe the performance counters.
• As the load gradually increases, insertion into the SQL database increases and the max throughput will approach the bounds of disk IO supporting the SQL database and transaction log. This will be reflected by the Database Queue % Full counter growing and the collector and database not able to reduce the queue fast enough. Record that system threshold and back off that number of ACS forwarders that are enabled at a time.
• Enable ACS forwarders in stages and groups that fall under the newly discovered system threshold until all desired ACS forwarders have been enabled in the target environment.
• A “cruising altitude” has now been reached and there should be few, if any, agents listed in the ACS Agents – Not Running view.

• In the Operations Console, click the Authoring wunderbar. In the Authoring navigation pane, select the Rules node.
• In the details pane, Change Scope to Look for“Autopilot” and then select the Autopilot – Enable ACS Forwarders on Servers target. Click the OK button.
• In the details pane, scroll down and right-click the Autopilot – Enable ACS forwarders on Servers rule and select Override > Override the Rule > For all objects of class: Autopilot – Enable ACS forwarders on Servers in the shortcut menu.
• In the Override Properties dialog box, select the Arguments checkbox and change the corresponding Override Value to be the ACS collector(s) FQDN; for example: //NoLogo “ACS1.contoso.com, ACS2.contoso.com”. Select the Enabled checkbox and change the corresponding Override Value to True.
• Click the OK button to commit the change and to enable autopilot.

• In the Operations Console, click the Authoring wunderbar. In the Authoring navigation pane, select the Rules node.
• Uncheck the Enabled checkbox on the previously configured override to the Autopilot – Enable ACS forwarders on Servers rule.
• Then re-enable once back at “cruising altitude”.