SCOM - Check for and alert on low space on mount points - Part 1

I have a number of sites still running Exchange 2010 and because of the buggy management pack there is often not a way to alert on the drive space of an exchange server mount point because sites have removed the management pack.

In searching for a solution, I came across a nice article that helped get me started here:

http://www.powershellneedfulthings.com/?p=36

I took that script and added some additional information so that I could generate both an error condition on low drive space as well as a recovery alert so SCOM knew when to close the alert itself.

In order for this script to run cleanly, you first need to run the following powershell command on any server you want to run mout-point checks on:

new-eventlog -LogName System -Source OpsHealthScript

Next, schedule the following script to run periodically on your server via task scheduler or whatever program you might use to run scripts on a recurring basis.

$TotalGB = @{Name="Capacity(GB)";expression={[math]::round(($_.Capacity/ 1073741824),2)}}
$FreeGB = @{Name="FreeSpace(GB)";expression={[math]::round(($_.FreeSpace / 1073741824),2)}}
$FreePerc =    @{Name="Free(%)";expression={[math]::round(((($_.FreeSpace / 1073741824)/($_.Capacity / 1073741824)) * 100),0)}}
$Check = @{Name="Failed";expression={[math]::round(((($_.FreeSpace / 1073741824)/($_.Capacity / 1073741824)) * 100),0) -lt 10}}

 function get-mountpoints {
 $global:volumes = Get-WmiObject -computer $server win32_volume | Where-object {$_.DriveLetter -eq $null}
 $global:volumes | Select SystemName, Label, Capacity, FreeSpace, $Check | Format-Table -AutoSize
        }

$servers = [system.environment]::MachineName
foreach ($server in $servers){
get-mountpoints
}
$Flag = $global:volumes | Select $Check
If ($Flag -match "True") {
Write-EventLog -logname System -source OpsHealthScript -eventID 500 -Entrytype Error -message 'One or more Exchange mount points are below 10% free space.'
}
Else {
Write-EventLog -logname System -source OpsHealthScript -eventID 800 -Entrytype Information -message 'Exchange mount points are healthy.'
}

You can access the file here: mount-point-space.ps1

Once this is scheduled, you can setup alerting in a couple of ways, one is a simple alert detection within SCOM to detect Error 500 in the system event log, the other is a correlated alert detection so that each alert generated by the script does not create a new alert within SCOM. I'll go over that in additional detail.

Operations Manager Performance Trending with Excel - No SQL Required

A powerful tool for administrators is to trend data to troubleshoot performance problems and forecast future resource needs. In the past, I've run SQL queries but needed a way to instruct support staff on a basic means to accomplish the same tasks right from the console. Thankfully, Operations Manager and Excel allow just that.

Let's get started.

Fire up the Operations Manager Console to the Monitoring section, then open the Windows Computers view (or any section where you access the computer health view, such as SQL)

Find a server you're interested in or simply select one from the list.

 

After selecting the system, select the Performance View under the Navigation pane of the task panel on the right side of the management console.

 

 

Once the Performance View window comes up, in the Performance Actions pane on the right side of the console, change your time frame via Select Time Range, selecting a meaningful period, such as two weeks or longer.

 

 

 

 

Now at the bottom of the performance monitor screen, select a counter you're interested. I'll use Percent Memory Used for this example.

When selected, a graph should display such as the following:

 

  

Going back to the Performance Actions pane, select Copy Data to Clipboard

Open up notepad and past the contents, which should look similar to the following:

Save the file with an xml extension

 

 

  

Now open Excel, select the Data tab and select From Other Sources -> From XML Data Import

 

Select the XML file created earlier; accept the import defaults when prompted

 

This should populate the Excel spreadsheet with an X and Y column. The first column is the date/time stamp and the Y column is the performance data.

 

 

Select all of the Y data and with it highlighted, select the Insert tab -> Line -> 2-D Line to generate a graph.

 

This should yield a graph in Excel such as the following:

 

Right-click on the graph line and select Add Trendline

 

Generally, accepting the default will paint a trendline  that is helpful for finding issues such as a memory leak or consistent data usage on a hard drive. However, you can play around with the trend to perform longer-term forecasts. I added 50 periods to the end of my trend line to see how memory might look in the future after my data set.

 

 

Graph results with the trendline:

 

 

The line extends a bit beyond the graph data and shows an overall flat trend on memory utilization. If the server had a memory leak, as an example, the graph might trend steadily upwards like this:

 

There you have it, a simple but powerful tool for analyzing data recorded in SCOM without a lot of effort.

AD Site Availability Degraded / AD Site Performance Health Degraded

After deploying the Active Directory Management Packs, we had a domain controller start alert spewing. I had not come across anything out there that really dealt with the alert; the warning from this type of event was not in eventid.net either. But it's all figured out now and here is the solution to the perplexing problem I encountered.

You could also title this, "How to Perform an Online/Offline Defragmentation of your Health Service Store in System Center".

Problem Description:

First, the SCOM console began to fill up with AD Site Availability Health Degraded and AD Site Performance Health Degraded critical alerts from the Active Directory Management Packs.

On the offending domain controller, I observed the following Application event log spewing:

 



 

The contents of the warning were as follows:

Troubleshooting:

Initially what I suspected was that I had an application or process going bonkers on the server, taking up memory and causing the SCOM agent to malfunction or be starved of resources. I loaded the Systernals Process Monitor utility to see what was happening when these events fired off, since typically it only took a few minutes in between each event. What was captured was a significant amount of file activity from the Health Service to

C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\HealthServiceStore.edb . Essentially, there was no other process at the time of these warnings or corresponding alerts in the System Center Management Console that could account for issues on the system.

 

 

 

With the smoking gun being the Health Service Database, I performed some quick online maintenance from within the console to start.

 

In the Operations Manager Console, I started by browsing to the Operations Manager folder, then Agent Details and selecting the Agents by Version view.

 

 

Selecting the offending computer brought up the Health Service Tasks I could perform, Start Online Store Maintenance, being the one I was looking for.

 

Final Solution:

Unfortunately, the online store maintenance was not adequate enough to remediate the errors and warnings I was encountering so I opted for an offline defragmentation of the Health Service Store database. Perform the following if local warnings persist on the client system.

 

• Login to the offending client system via console or RDP
• Open an administrative command prompt
• Change directory to "C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store"
• From the service console (services.msc) or from command prompt (net stop “Microsoft Monitoring Agent”), stop the Microsoft Monitoring Agent service
• Run esentutl /r edb (without this, you likely won't be able to perform a defragmentation)
• Next, run esentutl /d HealthServiceStore.edb

When this completed, my HealthServiceStore.edb file went from 174MB to 27Mb and both the warnings in the local Application event log and the critical health alerts in the System Center Operations Manager Console went away.

SCOM 2012 Failed Accessing Windows Event Log with Veeam Management Pack

Noticed during a routine health check that our two Management Servers were showing a warning state. Error read as "Failed Access Windows Event Log" <management server 1> (Health Service).

Error details show the following:

The Windows Event Log Provider is still unable to open the Veeam Collector event log on computer 'management server 1'. The Provider has been unable to open the Veeam Collector event log for 720 seconds. Most recent error details: The specified channel could not be found. Check channel configuration. One or more workflows were affected by this. Workflow name: many Instance name: many Instance ID: many Management group:

We have the Veeam management pack for SCOM loaded and sure enough, this appears to be a documented issue on the Veeam knowledge base.

http://www.veeam.com/kb1496#/kb1496

Windows 2012 WMI Hotfix

Had a 2012 Server that was being monitoring by System Center lock up on us today. Suspect a WMI leak. Hotfix deployment, engage!

http://support.microsoft.com/kb/2790831/en-us

SCOM 2012 Powershell - Retrieving a List of Computers in a Group

Had to search for a batch file that is on one of the many SQL servers we have in the environment. First inclination was, let me pull the systems from SCOM since it has all our SQL servers.

Poked around the interwebs a while and noticed a lot of scripts had references to 2007 commands that hadn't been updated to 2012. Here's the basic steps taken to get my group of SQL servers. You could perform the same task on pretty much any group in the same manner.

• Open the Operations Manager Shell powershell console

• Type in : Get-SCOMGroup

• Search for the group you want to retrieve members from

• Now type in: $Group = Get-SCOMGroup |  where {$_.DisplayName -eq "SQL Computers"} (or insert the group your looking for instead of SQL Computers")

• Next, type in: $Members = $Group.GetRelatedMonitoringObjects()

 

• Now, you can simply type: $Members

 

• Or, pipe the command out to a file: $Members | Sort DisplayName | FT DisplayName | out-file C:\Scripts\Servers.txt

 

Automated Discovery and Troubleshooting of Gray State Systems in System Center 2012 (Part-1)

Recently come across a rash of clients and internal systems at the office where monitored devices, for whatever reason, have gone into a gray state. I needed a way to quickly discover these systems, and ideally, run a script that would take some basic actions to remediate or troubleshoot these agents. In this first post, I'll give the full code necessary to get the gray agent discovery running. In the second post, I'll give a powershell script that detects the grayed out agents, shuts down the HealthService, clears the agent health directory, and then turns the HealthService back on automatically.

I came across three lines of code in the following blog, which got me pointed in the right direction. However, the code did not work correctly as provided.

http://www.bictt.com/blogs/bictt.php/2011/05/27/scom-trick-14-troubleshoot-grey

$WCC = get-monitoringclass -name "Microsoft.SystemCenter.Agent"
$MO = Get-MonitoringObject -monitoringclass:$WCC | where {$_.IsAvailable -eq $false}
$MO | select DisplayName

With just that code, I would receive the following error screen:

If you update the code to include the following path and connection to your system center server, the code will function properly. Running this should spit out a list of computers with a gray state in the agent status. This code should all be included in your powershell script:


$RMSFQDN = "<your SCOM managment server FQDN>"
$Name = "Microsoft.EnterpriseManagement.OperationsManager.Client"
$ModuleLoaded = Get-Pssnapin $Name -ErrorAction SilentlyContinue
If (-not $ModuleLoaded)
{
add-pssnapin "Microsoft.EnterpriseManagement.OperationsManager.Client";
}
New-ManagementGroupConnection -ConnectionString $RMSFQDN
Set-Location "OperationsManagerMonitoring::";

$AgentClass = get-monitoringclass -name:Microsoft.SystemCenter.Agent
$MO = Get-MonitoringObject -monitoringclass:$AgentClass | where {$_.IsAvailable -eq $false}

$MO | select DisplayName

Also, review this link for a comprehensive list of WMI hotfixes for various platforms:

http://support.microsoft.com/kb/2591403

Updated 11-15-2013: Review this link for agent based system hotfixes: http://support.microsoft.com/kb/2843219

Powershell script to stop system maintenance mode

In the event you want to pull the computer out of maintenance mode for some reason, here is the script for ending the maintenance mode. The command line prompts could be hard coded as variables or you could add a $params statement to the front of the script to feed in a variable.

$RMSFQDN = Read-Host "Enter the FQDN of your management server"

$Name = "Microsoft.EnterpriseManagement.OperationsManager.Client"

$ModuleLoaded = Get-Pssnapin $Name -ErrorAction SilentlyContinue

If (-not $ModuleLoaded)
{
add-pssnapin "Microsoft.EnterpriseManagement.OperationsManager.Client";
}

New-ManagementGroupConnection -ConnectionString $RMSFQDN
Set-Location "OperationsManagerMonitoring::";

$computerClass = get-monitoringclass -name:Microsoft.SystemCenter.ManagedComputer
$computerPrincipalName = Read-Host "Enter the FQDN computer name for ending maintenance:"
$computerCriteria = "PrincipalName='" + $computerPrincipalName + "'"
$computer = get-monitoringobject -monitoringclass:$computerClass -criteria:$computerCriteria


if($computer -eq $null)
{
$unixClass = get-monitoringclass -name "Microsoft.Unix.Computer"
$monObject = Get-MonitoringObject -monitoringclass:$unixClass
$computer = $monObject | where {$_.displayname -eq $computerPrincipalName}
}
ELSE
{
$computerClass = get-monitoringclass -name:Microsoft.Windows.Computer
$computer = get-monitoringobject -monitoringclass:$computerClass -criteria:$computerCriteria
}


if($computer.InMaintenanceMode -eq $true)
{
"Stopping maintenance mode for: " + $computerPrincipalName
$computer.StopMaintenanceMode([DateTime]::Now.ToUniversalTime(),[Microsoft.EnterpriseManagement.Common.TraversalDepth]::Recursive);
}
ELSE
{
"The computer " + $computerPrincipalName + " is not in maintenance mode."
}

Comprehensive Powershell Script for System Maintenance Mode

The script I posted yesterday was a quick and dirty script to put a windows host into maintenance mode. Problem is, that script doesn't work on agentless systems or Linux/Unix, so I took care of that. The following script will work for windows hosts, agentless hosts and linux/unix. The caveat is you have to enter the fully qualified domain name of the host. Alternatively, you could alter the script to append your DNS suffix to the host name to save some key strokes. Enjoy!

$RMSFQDN = Read-Host "Enter the FQDN of your Management Server"
$Name = "Microsoft.EnterpriseManagement.OperationsManager.Client"

$ModuleLoaded = Get-Pssnapin $Name -ErrorAction SilentlyContinue

If (-not $ModuleLoaded)
{
add-pssnapin "Microsoft.EnterpriseManagement.OperationsManager.Client";
}

New-ManagementGroupConnection -ConnectionString $RMSFQDN
Set-Location "OperationsManagerMonitoring::";

$startTime = [System.DateTime]::Now
$Hours = Read-Host "Enter the number of hours to put into maintenance:"
$endTime = $startTime.AddHours($Hours)
$comment = "Computer Maintenance"



$computerClass = get-monitoringclass -name:Microsoft.SystemCenter.ManagedComputer
$computerPrincipalName = Read-Host "Enter the FQDN computer name"
$computerCriteria = "PrincipalName='" + $computerPrincipalName + "'"
$computer = get-monitoringobject -monitoringclass:$computerClass -criteria:$computerCriteria


if($computer -eq $null)
{
$unixClass = get-monitoringclass -name "Microsoft.Unix.Computer"
$monObject = Get-MonitoringObject -monitoringclass:$unixClass
$computer = $monObject | where {$_.displayname -eq $computerPrincipalName}
}
ELSE
{
$computerClass = get-monitoringclass -name:Microsoft.Windows.Computer
$computer = get-monitoringobject -monitoringclass:$computerClass -criteria:$computerCriteria
}

if($computer.InMaintenanceMode -eq $false)
{
"Putting " + $computerPrincipalName + " into maintenance mode"
New-MaintenanceWindow -startTime:$startTime -endTime:$endTime -comment:$comment -monitoringObject:$computer
}

Scripting the Maintenance Window for a System through Powershell

Enclosed is a powershell script I put together from a number of examples out there. I ran into some overly complex scripts or ones that didn't quite work right when trying to find something quick and easy to use.

Ultimately, in our environment, I hard coded many of the variables to make the process quick and easy (Basically everything but the computer name). For the purposes of this script, it prompts for everything but the time window. I have removed the comments for the time being until I can get it to show up nicely in the blog.

$RMSFQDN = Read-Host "Please enter the FQDN of your management server:"
add-pssnapin "Microsoft.EnterpriseManagement.OperationsManager.Client";
New-ManagementGroupConnection -ConnectionString $RMSFQDN
Set-Location "OperationsManagerMonitoring::";

$startTime = [System.DateTime]::Now

$Hours = 1

$endTime = $startTime.AddHours($Hours)

$strComputerName = Read-Host "Enter the computer name to be put into maintenance mode:"

$objComputer = Get-Agent | Where-Object {$_.Name -match $strComputerName}

$comment = Read-host "Please provide a comment for putting the system in maintenance mode:"

$objComputer.HostComputer | New-MaintenanceWindow -StartTime:$startTime -EndTime:$endTime -Comment:$Comment

Duplicate name on network warning in SCOM

Came across this lovely gem recently. Troubleshooting steps didn't really help, at all.

A duplicate name has been detected on the TCP network. The IP address ofthe machine that sent the message is in the data. Use nbtstat -n in acommand window to see which name is in the Conflict state.

Servers in this case were in the DMZ, weren't in a domain and definitely did not have duplicate names. They did have multiple network adapters, but that shouldn't have caused the issue. Finally realized, the systems had been converted (PtoV) into our virtual enviornment from physical.

Dropped to a command prompt and ran "set devmgr_show_nonpresent_devices=1" to ensure device manager would show all hidden devices. From there, I ran devmgmt.msc to open device manager and then from the toolbar, I selected the "Show hidden devices" option. Going through, the old network adapters were still in there. After removing the adapters and rebooting, the warning did not come back.

http://support.microsoft.com/kb/315539 explains how to show hidden devices in windows.

Removing SCOM 2012 Management Packs Through Powershell

I got a little overzealous in importing management packs and found myself with a number of management packs that I did not want in my library. Going through one by one and selecting delete from the management window was going to be a nightmare. Enclosed is a powershell script I found and used to remove the various packs I imported. In this case, the management packs had country code extensions, such as .RUS or .KOR, that I didn't need for my installation. I will say, while this reduces the manual nightmare, the process takes exceptionally long, especially when using multiple search strings.

http://technet.microsoft.com/en-us/library/hh545200.aspx

Get-SCManagementPack ?{ $_.name -match ".RUS" } Remove-SCOMManagementPack

In some cases, you might want to delete multiple types of management packs. I've included an example of one using the "logical or" operation.

Get-SCManagementPack ?{ $_.name -match ".CHT" -or ".JPN" } Remove-SCOMManagementPack