Search This Blog

Showing posts with label "Troubleshooting". Show all posts
Showing posts with label "Troubleshooting". Show all posts

Thursday, January 8, 2015

Operations Manager Performance Trending with Excel - No SQL Required

A powerful tool for administrators is to trend data to troubleshoot performance problems and forecast future resource needs. In the past, I've run SQL queries but needed a way to instruct support staff on a basic means to accomplish the same tasks right from the console. Thankfully, Operations Manager and Excel allow just that.

Let's get started.

Fire up the Operations Manager Console to the Monitoring section, then open the Windows Computers view (or any section where you access the computer health view, such as SQL)


Find a server you're interested in or simply select one from the list.

 
After selecting the system, select the Performance View under the Navigation pane of the task panel on the right side of the management console.
 
 
Once the Performance View window comes up, in the Performance Actions pane on the right side of the console, change your time frame via Select Time Range, selecting a meaningful period, such as two weeks or longer.
 
 
 
 
Now at the bottom of the performance monitor screen, select a counter you're interested. I'll use Percent Memory Used for this example.


When selected, a graph should display such as the following:
 
  
Going back to the Performance Actions pane, select Copy Data to Clipboard

Open up notepad and past the contents, which should look similar to the following:


Save the file with an xml extension
 
 
  
Now open Excel, select the Data tab and select From Other Sources -> From XML Data Import
 

Select the XML file created earlier; accept the import defaults when prompted
 

This should populate the Excel spreadsheet with an X and Y column. The first column is the date/time stamp and the Y column is the performance data.
 
 
Select all of the Y data and with it highlighted, select the Insert tab -> Line -> 2-D Line to generate a graph.
 

This should yield a graph in Excel such as the following:
 

Right-click on the graph line and select Add Trendline
 

Generally, accepting the default will paint a trendline  that is helpful for finding issues such as a memory leak or consistent data usage on a hard drive. However, you can play around with the trend to perform longer-term forecasts. I added 50 periods to the end of my trend line to see how memory might look in the future after my data set.
 
 
Graph results with the trendline:
 
 
The line extends a bit beyond the graph data and shows an overall flat trend on memory utilization. If the server had a memory leak, as an example, the graph might trend steadily upwards like this:

 
There you have it, a simple but powerful tool for analyzing data recorded in SCOM without a lot of effort.

Tuesday, December 23, 2014

AD Site Availability Degraded / AD Site Performance Health Degraded

After deploying the Active Directory Management Packs, we had a domain controller start alert spewing. I had not come across anything out there that really dealt with the alert; the warning from this type of event was not in eventid.net either. But it's all figured out now and here is the solution to the perplexing problem I encountered.

You could also title this, "How to Perform an Online/Offline Defragmentation of your Health Service Store in System Center".

Problem Description:

First, the SCOM console began to fill up with AD Site Availability Health Degraded and AD Site Performance Health Degraded critical alerts from the Active Directory Management Packs.

AD Site Availability Health Degraded and AD Site Performance Health Degraded

On the offending domain controller, I observed the following Application event log spewing:
 

 
The contents of the warning were as follows:
HealthService (1704) A significant portion of the database buffer cache has been written out to the system paging file. This may result in severe performance degredation. See help link for complete details of possible causes. Log Name: Application | Source: ESENT | Event ID: 906

Troubleshooting:

Initially what I suspected was that I had an application or process going bonkers on the server, taking up memory and causing the SCOM agent to malfunction or be starved of resources. I loaded the Systernals Process Monitor utility to see what was happening when these events fired off, since typically it only took a few minutes in between each event. What was captured was a significant amount of file activity from the Health Service to
C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\HealthServiceStore.edb . Essentially, there was no other process at the time of these warnings or corresponding alerts in the System Center Management Console that could account for issues on the system.
 
 
SCOM HealthService | ReadFile | C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\HealthServiceStore.edb
 
With the smoking gun being the Health Service Database, I performed some quick online maintenance from within the console to start.
 
In the Operations Manager Console, I started by browsing to the Operations Manager folder, then Agent Details and selecting the Agents by Version view.
Management Console Tree -> Operations Manager -> Agent Details -> Agents By Version
 
 
Selecting the offending computer brought up the Health Service Tasks I could perform, Start Online Store Maintenance, being the one I was looking for.
Management Console Health Service Task for Health Service Database Maintenance | Start Online Store Maintenance
 
Final Solution:

Unfortunately, the online store maintenance was not adequate enough to remediate the errors and warnings I was encountering so I opted for an offline defragmentation of the Health Service Store database. Perform the following if local warnings persist on the client system.
 
  • Login to the offending client system via console or RDP
  • Open an administrative command prompt
  • Change directory to "C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store"
  • From the service console (services.msc) or from command prompt (net stop “Microsoft Monitoring Agent”), stop the Microsoft Monitoring Agent service
  • Run esentutl /r edb (without this, you likely won't be able to perform a defragmentation)
  • Next, run esentutl /d HealthServiceStore.edb
Running esentutl /d HealthServiceStore.edb in order to compact and defragment the health service database after log spewing occurred from loading the Active Directory management packs

When this completed, my HealthServiceStore.edb file went from 174MB to 27Mb and both the warnings in the local Application event log and the critical health alerts in the System Center Operations Manager Console went away.

Wednesday, February 22, 2012

Duplicate name on network warning in SCOM

Came across this lovely gem recently. Troubleshooting steps didn't really help, at all.

A duplicate name has been detected on the TCP network. The IP address ofthe machine that sent the message is in the data. Use nbtstat -n in acommand window to see which name is in the Conflict state.

Servers in this case were in the DMZ, weren't in a domain and definitely did not have duplicate names. They did have multiple network adapters, but that shouldn't have caused the issue. Finally realized, the systems had been converted (PtoV) into our virtual enviornment from physical.

Dropped to a command prompt and ran "set devmgr_show_nonpresent_devices=1" to ensure device manager would show all hidden devices. From there, I ran devmgmt.msc to open device manager and then from the toolbar, I selected the "Show hidden devices" option. Going through, the old network adapters were still in there. After removing the adapters and rebooting, the warning did not come back.

http://support.microsoft.com/kb/315539 explains how to show hidden devices in windows.