Rate, Volume, Utilization, and Parsecs

But wait, the parsec is not a unit of time, but a unit of distance! Wait, what? All arguments aside about how the Millennium Falcon could make the Kessel run in a shorter distance through enormous gravitational shears, knowing your unit is extremely important.

I work in network monitoring and one of the main reports my tools provide measures how much an interface is used. Because the tool is better than poo, it presents the utilization in several different units. First, let's review the units. Each of these units are SI units, so standard SI prefixes apply when talking about larger bases of the base unit:

• Bytes - measures the total number of octets that were transmitted (or received depending on p.o.v.)
• Bits per second - measures the number of 0's and 1's that were transmitted (or received depending on p.o.v.) in a single second.
• Percent utilization (%) - measures the percentage of a period of time that the interface was transmitting (or receiving).

Let's break it down.

---

Bytes

This one is pretty simple and is referred to as VOLUME. It's simply the total number of Bytes transmitted (or received) during the measurement window. An SNMP polling station would poll the octet counter at a regular interval. Every time the octet counter is polled, the delta between the previous poll results and the current poll results represents the total number of Bytes during the measurement interval.

V = B₁ - B₀

Polling too frequently will result in small values. Whenever rollups happen, the individual data points should be summed (integrate over the rollup interval). As long as rollups are done that way, the poll rate is less consequential.

Rollover is accounted for by assuming that a lower number than the previous measurement is caused by rollover and the new measurement (measured from 0) is added to whatever remained between the previous measurement and the max limit of the counter.

Layman's example

This is similar to tracking how many miles a car travels. You simply take a reading of the odometer before beginning a trip and another at the end of the trip. The difference is the total miles the trip entailed. You could take readings more often. You'd just need to add up all your measurements at the end of the trip to get the total for the trip.

---

Bits per second

Bits per second is a simple count measured over a unit of time, making it a RATE. It counts the number of bits that went through the interface, then normalizes the count over a standard unit of time, the second. It is calculated like this:

R = (Δ bits) / (Δ time)

That is, you take the total number of bits and divide it over the total time of the measurement. This is usually done through SNMP by looking at the octet counters. The NMS will poll the sysUpTime and the octet counters at a certain time (T₀ and B₀). It will then poll the sysUpTime and octet counters at some other time in the future (T₁ and B₁). The RATE is calculated by dividing the difference between these two measurements (and adjusting the octet counters to get it into bits instead of bytes 8 bits = 1 Byte):

R = 8 (B₁ - B₀) / T₁ - T₀

The resulting unit is bits/second and represents an average of the number of bits transmitted per second over the measurement interval (T₁-T₀). When doing the rollup, average is the most common descriptor. In addition, min, max, standard deviation, variance, and 50^th, 75^th, and 90^th percentiles would be useful.

If you're already gathering VOLUME, you'll notice that B₁ - B₀ used in the RATE calculation comes from the volume calculation. That's on purpose and is why it is said that RATE is derived from the VOLUME measurement. In fact, if the polling interval is fairly regular, the rate can be said to be approximately linearly proportional to the volume.

Layman's example

This is not really any different than measuring the speed of your car while on a trip. You take a reading of the odometer and the clock at the beginning of the trip and again at the end of the trip. The difference in miles, divided by the total time of the trip (in hours in this case) will give you an average speed in mph. You could increase the resolution of your measurements by taking a reading and performing the calculation every 5 minutes. This would give you a data point describing the average speed for every 5 minutes of your trip.

---

Percent Utilization

Percent UTILIZATION measures how much capacity is used and is reported in percentage of the total capacity available. This is calculated by dividing the current RATE by the total rate capable by the interface. Alternatively, it could be calculated by dividing the VOLUME by the total volume capability of the interface. The latter requires a bit more derivation, so most use the former.

This metric requires knowledge about the interface's capabilities. This is usually obtained by polling the bandwidth statement (ifspeed) of the interface (1.3.6.1.2.1.2.2.1.5), which is in bits per second (bps). Once obtained, the percent UTILIZATION can be calculated like this:

U = 8 (B₁ - B₀) / T₁ - T₀ / ifSpeed

You may notice that a part of this formula looks the same as the RATE calculation. It is. Simplifying the formulas:

U = 8 (B₁ - B₀) / T₁ - T₀ / ifSpeed * 100
R = 8 (B₁ - B₀) / T₁ - T₀
U = R / ifSpeed * 100

Since the UTILIZATION formula involves dividing a rate (in bps) by a speed (in bps), the result is unitless. This means that the unit can be thought of as % (percentage). Rollups of UTILIZATION should be treated the same way as rollups for RATE. You should also notice that the percent utilization should be linearly proportional to the rate, given a constant bandwidth capacity of the interface.

Layman's example

This calculation is similar to calculating how close a driver is to the speed limit. By dividing the current speed (derived using the formulas above for speed) by the total allowable speed, you can calculate what percentage of the limit the car is currently traveling. When driving, moving at 100% of the speed limit is actually good. You are actually making the most of the available resource. The only time 100% utilization is a problem is when you need to do something else with that speed (i.e. other cars on the road not travelling at the same speed). The same actually holds true for networking. Utilization of 100% is not bad until you need some percentage of those resources for another task.

NQBackup

UPDATE 6/9/2015: Version 1.7 now released. This update adds standalone support. Since CA is including newer versions of MySQL in their products, DBToolv3 is no longer going to work. This change allows you to specify to use MySQLDump instead of DBToolv3. Essentially, you unremark line 15 and remove/remark line 14. If I get enthusiastic about it, I may update the script to allow a switch from the command line to specify which method to use. I'm just not there yet.
UPDATE 2/10/15: Version 1.6 now released. This update changes the way harvesters and DSAs are backed up, by only backing up the ReaperArchive, ReaperArchive15, and HarvesterArchive directories to a single directory (no redundant rolling backups). It only backs up files that have the archive bit set, so before running it the first time, set the archive bit for all the files in those directories. I also fixed the date naming method so it's YYYYMMDD instead of YYYYDDMM. I also added timestamping to the log so you know how long it takes to perform the file backups vs. the database backups.
UPDATE 2/27/14: Version 1.5 now released. This version doesn't have too many changes. I just added the lines below that allow the NFA mess of data files to be backed up along with everything else. This one script can still be used on any product. However, when running on a Harvester or DSA, extra commands backup the data files.
The syntax for running the tool hasn't changed since 1.4 (but 1.4 introduces some major changes), so you should be able to drop the script in place without changing any scheduled tasks.

nqbackup.bat <dbname> <num_backups_to_keep>

Remember, if you need a reminder how to run the tool, just run it without any arguments (or just double click it from Windows Explorer).

Custom Device Polling in NetVoyant

This is a presentation I gave years ago but the recording on the community has been lost. So, I recorded it again and have posted it here.

UCDavis MIB for Monitoring Linux Memory

I discovered quite some time ago that the Net-SNMP agent on my RaspberryPi doesn't report memory utilization in the hrstorage MIB like I would expect it to.  It's not wrong, however, the value doesn't match up with what's actually being used on the device.  The reason for that is that Linux can use the system RAM for several things: processes, shared memory for processes, buffers, and disk cache.  When most people ask how much memory is in use, they are asking for how much memory is in use by the processes.  That's the value you get on the second row under the 'used' column of the output of the free command:
total used free shared buffers cached Mem: 497504 486784 10720 0 55596 397996 -/+ buffers/cache: 33192 464312 Swap: 102396 0 102396
However, the OIDs in the hrstorage MIB actually return the value from the first row of the 'used' column.  The problem is that both of these numbers represent memory utilization.  The first row shows the total of the processes, shared, buffers, and cache.  That's the total amount of memory that's in use on the system.  However, this isn't the value most people associate with memory utilization.

In order to get the correct value, there are two options.  The first doesn't work with NetVoyant, but it doesn't use additional MIBs or OIDs to get the data.

Since the shared, buffer, and cache memory is reported in the hrstorage table, you can simply take hrStorageSize of the Physical Memory row (hrStorageType==1.3.6.1.2.1.25.2.2), and subtract the hrStorageUsed from the shared, buffer, and cache rows (hrStorageType==1.3.6.1.2.1.25.2.1.1).  Since NetVoyant can't use values from other poll instances in an expression, it won't work in NV.

Side Note: This may be possible by creating a single expression that results in the positive value of hrStorageSize when the hrStorageType is .2 and a negative value of hrStorageUsed when hrStorageType is .1.  The sum of that expression for all the .1 and .2 poll instances should give you the total used memory.  However, since the sum could only be done in a view in the web GUI, it would only work for reporting and not thresholding/alarming.

The second option is to use the UCDavis MIB.  The NetSNMP agent does populate the UCDavis tables, so any of the values there can be polled.  The problem is that there's no real clear documentation on which OIDs give you which values when compared to the output of the free command.  Here's the mappping:
total used free shared buffers cached Mem: A B C D E F -/+ buffers/cache: G H Swap: I J K
Given the output above, here are the OIDs or combinations you need to calculate the values:

1. memTotalReal
2. memTotalReal - memAvailReal
3. memAvailReal
4. memShared
5. memBuffer
6. memCached
7. memTotalReal - memAvailReal - memShared - memBuffer - memCached
8. memAvailReal + memShared + memBuffer + memCached
9. memTotalSwap
10. memSwapError
11. memAvailSwap

Given this, it should be pretty easy to create a dataset to poll memory.  Just remember, these OIDs are in units of KB, so if you want it in Bytes so that NV automatically scales (to KB, MB, GB, TB, etc.) you'll need to multiply each one by 1024.  Obviously, if you're calculating % utilization, you don't need to multiply both the numerator and the denominator by 1024.  You will need to multiply by 100 to get the ratio to a scale of 0-100%.

Using Distributions to show Performance of Multiple Objects on a Time Scale

Many people building custom views in NV will no doubt build one of two types of views: Details Trend or Management TopN.  Unfortunately, this bypasses some of the cooler views like the distribution views.  Consider this scenario: I have multiple third party devices and the manufacturer has provided a special MIB to monitor CPU utilization (instead of doing the smart thing like publishing their CPU statistics into the hrprocessor or UCDavis MIB OIDs).  So, I now have the opportunity to build a custom dataset to pull in the CPU utilization for these devices.  (Side note, i should probably republish my instructions on how to build a custom dataset.)
After I build the dataset, I'll start building my views.  Let's suppose that the vendor has only provided the CPU utilization as an average of all the CPUs on the device or that the device will only ever have one CPU.  The end result is that there is only one poll instance per device for that dataset.  This means that I'll only really build views on the device level and configure the views to drill down to the device page instead of the poll instance page.  After building the appropriate trends on the device page, I'd go to an overview page and build a table or bar chart to show the devices with the highest CPU utilization.  All of this is great and normal and is what most people do when building views for this kind of data.
The problem with stopping here is that there is no way to look at multiple devices over a period of time and see how the devices were performing within that timeframe.  The reason for this is that a TopN table or bar chart will display the rollup (usually the average) of the metric within the timeframe.  In the case of my custom dataset, I'd see the average CPU utilization over the last hour, last day, last week, etc.  This is ok as long as I pick one of the standard timeframes.  Notice what happens when you pick last 4 hours in NPC.  A table or bar chart will only do last hour.  That's because NV hasn't pre-calculated rollups on a 4-hour basis.  So, it becomes important to show the performance of the metric over time showing the values within the timeframe, be it a standard rollup period or not.
That's where distribution views can help.  While they don't necessary show the value of each one of the poll instances analyzed, they do categorize the metric into groups.  For example, I could build a distribution view to group the metrics like: 0-25%, 25-50%, 50-75%, 75-95%, and over 95%.  In this case, NPC would look at all the data during the timeframe (if last hour with 5 minute polling, it will look at 12 data points for each poll instance included in the context) and categorize each data point into one of the buckets I've defined.  The end result is a trend plot over time showing how many devices are in which buckets for each point in time.

Users need to be instructed in the proper way to interpret the view.  If the view is setup properly, the undesirable buckets will have more extreme colors (reds and oranges).  When a user sees a time period in which a larger number of devices are in the undesirable buckets, they should understand that a large number of devices has experience higher CPU utilization.  If 10 devices' CPU utilization goes from 20% to 60%, the bars before the increase will show 10 devices in the 0-25% bucket while the bars after the increase will show 10 devices in the 50-75% bucket.  NPC also calculates the percentage of total devices in each bucket.  So, if half of my devices are in the 50-75% range, a mouseover will reveal 50% in that bucket.
This visualization can be equated to creating a pie chart for each poll cycle.  If you look at one poll cycle for all the devices and created a pie chart with 5 slices, it would be easy to understand how many devices need attention.  Imagine taking the crust off the pie, stretching it out flat and stacking it next to the pie crusts for the other poll cycles in the period.
One disadvantage to the distribution charts is that they lack drill down.  So, while a distribution is good for a summary page, a table showing the rollups over the same timeframe will be helpful to identify which devices are experiencing the higher CPU utilization.  This table would allow drill down to the device page where the individual trend plot could be analyzed individually.  It could also be compared to the rest of the data being gathered by NV for the device.

NetVoyant Trap Processing

I decided to go through some of the content I've garnered over the years and make a few videos.  This one shows how NetVoyant goes through its processing of incoming traps.  Enjoy!

Interface Summary Table Ultimate Tweak

A while back, I did a major customization of the Top Least Interfaces Table in NPC.  This is a NetVoyant view that normally shows interface availability and utilization in and out of every interface.  There's no reason, however, that that table can't contain many more metrics.  That's essentially what I did with this customization.

In order to implement this run the following command on the NPC server:
mysql -P 3308 -D netqosportal -e "replace into control_properties (controlid, propertyname, propertyvalue) values (20651,'ColumnNames','avail,Utilization_In,__Deviation_In,Baseline_In,e95th_In,Utilization_Out,__Deviation_Out,Baseline_Out,e95th_Out,Average_Utilization,Errors_In,Discards_Out,p_ifIndex_property_value,p_ifSpeed_in_property_value,p_ifSpeed_out_property_value'),(20651,'ControlType','Interface'),(20651,'data.chartType','Table'),(20651,'DataSetName','ifstats'),(20651,'Description','Originally the \'Top Least Available Interfaces\' view, this view has been highly modified by Stuart Weenig to include other major interface statistics.'),(20651,'DisplayFormats','percent:avail|percent|percent|percent|percent|percent|percent|percent|percent|percent|packetrate|packetrate|string|rate|rate'),(20651,'DisplayNames','Availability,Utilization In,% Deviation In,Baseline In,95th In,Utilization Out,% Deviation Out,Baseline Out,95th Out,Average Utilization,Errors In,Discards Out,Index,Speed In,Speed Out'),(20651,'FieldNames','(ifuptime/ifduration)*100;in_ifutil;(in_ifutil-in_ifutil_bl)/in_ifutil_bl*100;in_ifutil_bl;pct_in_ifutil;out_ifutil;(out_ifutil-out_ifutil_bl)/out_ifutil_bl*100;out_ifutil_bl;pct_out_ifutil;(in_ifutil+out_ifutil)/2;in_errorrate;out_discardrate;p_ifIndex.p'),(20651,'FieldNames2','roperty_value;p_ifSpeed_in.property_value;p_ifSpeed_out.property_value'),(20651,'OrderBy','Average_Utilization DESC'),(20651,'PropertyNames','Description,ifSpeed,ifType,Name,ifIndex,ifDescr,ifName,ifSpeed_in,ifSpeed_out'),(20651,'Title','Interface Utilization Summary');update controls set title='Interface Utilization Summary Table' where id=20651;"

That should do it. Now the default definition for that view should contain all the advanced metrics shown above. The view also has a new title when in the view list: "Interface Utilization Summary". The way you can know it'st he right one is by hovering over the view in the view list and it should pop up a description with my name in it.

This can also be applied directly to the NV view.  You can do it through the wizard, or just run the following command on the NV server:
mysql -P 3308 -D nms2 -e "replace into control_properties (controlid, propertyname, propertyvalue) values (20150,'ColumnNames','avail,Utilization_In,__Deviation_In,Baseline_In,e95th_In,Utilization_Out,__Deviation_Out,Baseline_Out,e95th_Out,Average_Utilization,Errors_In,Discards_Out,p_ifIndex_property_value,p_ifSpeed_in_property_value,p_ifSpeed_out_property_value'),(20150,'ControlType','Interface'),(20150,'data.chartType','Table'),(20150,'DataSetName','ifstats'),(20150,'Description','Originally the \'Top Least Available Interfaces\' view, this view has been highly modified by Stuart Weenig to include other major interface statistics.'),(20150,'DisplayFormats','percent:avail|percent|percent|percent|percent|percent|percent|percent|percent|percent|packetrate|packetrate|string|rate|rate'),(20150,'DisplayNames','Availability,Utilization In,% Deviation In,Baseline In,95th In,Utilization Out,% Deviation Out,Baseline Out,95th Out,Average Utilization,Errors In,Discards Out,Index,Speed In,Speed Out'),(20150,'FieldNames','(ifuptime/ifduration)*100;in_ifutil;(in_ifutil-in_ifutil_bl)/in_ifutil_bl*100;in_ifutil_bl;pct_in_ifutil;out_ifutil;(out_ifutil-out_ifutil_bl)/out_ifutil_bl*100;out_ifutil_bl;pct_out_ifutil;(in_ifutil+out_ifutil)/2;in_errorrate;out_discardrate;p_ifIndex.p'),(20150,'FieldNames2','roperty_value;p_ifSpeed_in.property_value;p_ifSpeed_out.property_value'),(20150,'OrderBy','Average_Utilization DESC'),(20150,'PropertyNames','Description,ifSpeed,ifType,Name,ifIndex,ifDescr,ifName,ifSpeed_in,ifSpeed_out'),(20150,'Title','Interface Utilization Summary');update controls set title='Interface Utilization Summary Table' where id=20150;"

NV Default Tweaks

To go along with my post about the default tweaks that I do to a vanilla SuperAgent (ADA) installation, I decided to go ahead and document my default tweaks for NetVoyant.  Note the disclaimer at the bottom of this page.  All of these tweaks should be done before the first discovery cycle begins.

1. Add discovery scopes by network, not individual IP address.  This is a hot topic, but I maintain that using networks is better than individual IP addresses, if only for the sake of administration.  If you've configured DNS and discovery properly (see point 5 below) IP address changes won't require any intervention.  If you'd rather keep a super tight grip on your stack, go right ahead.
2. Enable daily Periodic Discover: just a checkbox under discovery
3. Tweak SNMP Timeout: Change the timeout from 5 seconds to 2.  If it hasn't responded after 2 seconds, it's not going to respond after 5.
4. Enable Reachability Only Monitoring: If you want to monitor devices in scope but not SNMP capable, you can by only using ICMP.  Enable this by unchecking the box that says 'Ignore Non-SNMP Devices'.  You'll also need to go to Config>>Discovery>>Device Models and check the 'Enabled' checkbox on the 'NonSNMP Devices' model.
5. Update Device Naming: This one takes some thinking.  If you know you will have DNS entries for all of your devices, the best would be to let NV poll via FQDN (vs. polling by IP address).  That way, if your discovery scopes include networks instead of individual IP addresses you won't have to change anything in NV when the IP address of a device changes.  Since NV will be polling via FQDN and the new IP address is still in scope, NV won't know any different.  Set Default device name to 'DNS Name'.  If there isn't one, NV will poll via IP address.
6. Give NV more resources: Slide the resource usage slider up to its max.  If NV isn't the only thing on the server, do this carefully.
7. Disable Undesired Classes: Under Discovery>>Device Classes disable any device classes you won't want to monitor.  This is one way you can prevent NV from monitoring everything on your network even though you've added scopes by network.  I typically disable Printers and workstations.  You will need to keep an eye on any SNMP capable devices that show up in the other group.  This means NV doesn't know what class the device belongs to.  Right click the device and click change classification.  If you need a new class, come to Config>>Discovery>>Device Classes and create it.  After you make a classification change, make sure your undesired classes still say "No Device Models Enabled Upon Discovery'.
   Tip: when you're reclassifying devices, you can set the icon that gets used by the NV console when displaying the device.  This is only for the console, but it can make things easier to troubleshoot.  You can either use one of the built in images (found at D:\netqos\netvoyant\classes\redpoint\images) or store your own there (keep it to less than 20x20 pixels) by entering the image name (without the .gif) in the change classification dialog box.
8. Disable polling of the System Idle Process: If the Host Resources Software Performance (hrswrun) dataset is going to be used, setup a discovery rule called 'Default' with expression:
   

   hrSWRunName <> 'System Idle Process'

   It's also a good idea to go ahead and set the poll event severity to none.  Otherwise you'll get an alarm every time a process fails to poll.  This can be a good thing, since it indicates that a process has gone down.  However, if NV is polling a process that is being run by a user, when the user logs off, the process will disappear.  In fact, I usually go through and disable poll events for all datasets.  This should be done understanding what is lost when not getting poll events.
9. Disable Host Resource Device Table (hrdevice): Create a discovery rule called 'None' with expression:
   

   1==2

   If you've already discovered some/all of your devices, set the poll instance expiration to 0 and enable the 'None' discovery rule.  Then run a full rediscovery.  After that's done, disable polling and periodic discovery on that dataset.
10. Disable VMware datasets: You will only get data for these datasets if you own CA Virtual Assurance.  If you do, skip this step.  If you don't, disable polling and periodic discovery for VMware Datacenter Element (aimdc), VMware Host (aimhost), and VMware Virtual Machine (aimvm).
11. Disable NBAR and RMON2: if you have NBAR or RMON2 probes and want to poll them from NV, skip this step.  Otherwise, disable polling and periodic discovery for Protocol Distribution (NBAR) (nbarstats) and Protocol Distribution (RMON2) (protodist).
12. Disable polling of optical, removable, and floppy drives: Add a discovery rule to the Host Resource Storage (hrstorage) dataset called 'Default' with expression:
    

    hrStorageType NOT IN ('1.3.6.1.2.1.25.2.1.7','1.3.6.1.2.1.25.2.1.5')

    If you've already discovered some/all of your devices, set the poll instance expiration to 0 and enable the 'Default' discovery rule.  Then run a full rediscovery.  After that's done, set the poll instance expiration back to something reasonable like 28.
13. Disable polling of various interface types: Add a discovery rule called 'Default' with expression:
    

    ifInOctets+ifOutOctets<>0 AND ifType NOT IN (1, 18, 24, 134, 37, 100, 101, 102, 103, 104) AND ifSpeed<>0

    If you're curious about which interface types this excludes, look on the Config tab under Discovery>>Interface Types.
14. Enable Verbosity on the Topology service: Go to Services>>Topology and change the drop down from 'Normal' to 'Normal (Verbose)'.  There's no save button.  Turn this back to 'Normal' after NV is up and running and stable in production.
15. Disable Traps: If NV isn't going to be your trap handler, prevent stray traps from getting logged into the database by going to Services>>Traps and setting start mode to 'Manual'.  Then click 'Stop' to stop the service.
16. Configure your view options: Under the View menu, make sure everything is enabled.

That's it for now.  Make sure the discovery monitor is open and kick off discovery.  That should get you started.

Here's a picture:

Finding the data source for a particular device in NPC

Recently, we needed to know which data source was contributing to the report data for a particular device in NPC.  This was fairly easy to find out given a simple query:

mysql -P 3308 -D netqosportal -e "select a.itemname as Device, v6_ntoa(a.address) as Address, b.consolename as DataSource from dst_device as a, data_sources2 as b where a.sourceid=b.sourceid and itemname like '%devicename%' order by a.itemname;"

Simply replace devicename with the device name and execute this at a command prompt on the NPC server.  The result should look something like this:

+------------------------+-----------------+----------------------+
| Device                 | Address         | DataSource           |
+------------------------+-----------------+----------------------+
| center                 | 192.168.100.2   | NetVoyant            |
| nacogdoches            | 192.168.100.3   | NetVoyant            |
| nacogdoches            | 192.168.100.3   | ReporterAnalyzer     |
| houston                | 192.168.100.4   | ReporterAnalyzer     |
| houston                | 192.168.100.4   | NetVoyant            |
| dallas                 | 192.168.100.5   | ReporterAnalyzer     |
| dallas                 | 192.168.100.5   | NetVoyant            |
| sanfelipe              | 192.168.100.6   | ReporterAnalyzer     |
| sanfelipe              | 192.168.100.6   | NetVoyant            |
| austin                 | 192.168.100.7   | ReporterAnalyzer     |
| austin                 | 192.168.100.7   | NetVoyant            |
| elpaso                 | 192.168.100.8   | NetVoyant            |
| brownsville            | 192.168.100.9   | NetVoyant            |
| beaumont               | 192.168.100.10  | NetVoyant            |
| lufkin                 | 192.168.100.11  | NetVoyant            |
| ftworth                | 192.168.100.12  | NetVoyant            |
| ftworth                | 192.168.100.12  | ReporterAnalyzer     |
| tyler                  | 192.168.100.13  | ReporterAnalyzer     |
| tyler                  | 192.168.100.13  | NetVoyant            |
| henderson              | 192.168.100.14  | NetVoyant            |
| amarillo               | 192.168.100.15  | NetVoyant            |
| amarillo               | 192.168.100.15  | ReporterAnalyzer     |
| sanantonio             | 192.168.100.16  | NetVoyant            |
| bexar                  | 192.168.100.17  | NetVoyant            |
+------------------------+-----------------+----------------------+

It would be nice if NPC or the new CAPC had some kind of feature that showed the datasource(s) for a particular object on the device details page.

NetVoyant Device Inventory through ODBC

Continuing my effort to document the various ways I've used the ODBC connector for the NetQoS products, here's my next query and controls I've built and that I use in production.  Today's query comes from a need to export a list of all devices polled by NetVoyant to CSV.  This can usually be done by going to Inventory >> Devices and exporting to CSV from there.  However, that control has a limit of 5000 devices and we're monitoring more than that.  Luckily ODBC doesn't have that limitation.  Here's the SelectCommand and OdbcConnection String to put in the configuration.xml:

<selectcommand Name="DeviceInventory" Property="False" SupportedViewTypes="Table"> <![CDATA[ select convert(concat( '<a href=\'http://npc/npc/default.aspx?pg=', if(itemsubtypename='router','r',if(itemsubtypename='switch','sw',if(itemsubtypename='server','s','d'))), '&Deviceid=', itemid, '\'>', itemname, '</a>' ) using utf8) as dev_name, convert( v6_ntoa(address) using utf8) as address, convert( if(snmpversion=0,'Not SNMP Capable',snmpversion) using utf8) as snmpcapable, itemsubtypename, if( char_length(itemdesc)=0, null, if(char_length(itemdesc)<100,itemdesc, concat(left(itemdesc,100),'...')) ) as description from dst_device where sourceid=10; ]]> </SelectCommand> <odbcconnectionstring Name="NPC">Driver={MySQL ODBC 5.1 Driver};Server=192.168.10.10;Port=3308;Option=3;Database=netqosportal;User=netqos;Password=netqos;</OdbcConnectionString>
To create the view, run the following SQL commands against the NPC server:

INSERT INTO controls SELECT if(min(id)-1>0,min(id)-1,29999), 'NetVoyant Devices in NPC', 'Graph', 'NetVoyant-ODBC', '/nqWidgets/Integration/wicTable.ascx', 1, 0x200000, 0, 0, 0, 1, '|nc|g|', '', 'Y' FROM controls WHERE datasourcetype=0x200000; INSERT INTO control_properties (ControlID, Editable, PropertyName, PropertyValue) VALUES (-1, 'N', 'Title', 'NetVoyant Devices in NPC'), (-1, 'N', 'DisplayColumns', 'dev_name;address;snmpcapable;itemsubtypename;itemdesc'), (-1, 'N', 'DisplayFormats', 'string,string,string,string,string'), (-1, 'N', 'DisplayNames', 'Device Name,Address,SNMP Version,Type,Description'), (-1, 'N', 'ColumnWidths', ',,,,'), (-1, 'N', 'HorizontalAlignments', 'Left,Left,Left,Left,Left'), (-1, 'N', 'SortField', 'dev_name'), (-1, 'N', 'SortDirection', 'Asc'), (-1, 'N', 'Integration.Parameter.SelectCommandName', 'DeviceInventory'), (-1, 'N', 'Integration.DataType', 'SelectCommand'), (-1, 'N', 'Integration.Parameter.DateTimeFormat', 'yyyy-MM-dd HH:mm'), (-1, 'N', 'Integration.Parameter.TopN', '200'), (-1, 'N', 'Integration.Parameter.OdbcConnectionStringName','NPC'); UPDATE control_properties SET ControlID = (select min(id) from controls where datasourcetype=0x200000) WHERE ControlID = -1;

NetVoyant Duplicates through ODBC

Continuing my effort to document the various ways I've used the ODBC connector for the NetQoS products, here's my next query and controls I've built and that I use in production.  Today's query comes from a need to view duplicate devices and make it easy to eliminate the duplicates.  I had built a fairly extensive method to do this using Perl script and batch files.  It suffered from all the problems that most NPC browser views suffer from.  So, ODBC is a better way to accomplish this and I'm officially retiring that script.  Here's the SelectCommand and OdbcConnection String to put in the configuration.xml:

<selectcommand Name="NVDuplicates" Property="False" SupportedViewTypes="Table"> <![CDATA[ select convert(concat( '<a target=\'_blank\' href=\'http://NVMC/nv/default.aspx?pg=', if(a.class_id=30000,1,if(a.class_id=4,20004,if(a.class_id=7,30104,11004))), '&Deviceid=', a.device_id, '\'>', a.dev_alias, '</a>' ) using utf8) as Alias, a.dev_alias as aliassort, if(a.expiration_started=0,'-',from_unixtime(a.expiration_started, '%Y/%m/%d %H:%i')) as 'ExpiringSince', if(a.unresponsive_started=0,'-',from_unixtime(a.unresponsive_started, '%Y/%m/%d %H:%i')) as 'UnresponsiveSince', b.smgr_host as Poller, concat( '<a target=\'_blank\' href=\'//', b.smgr_host, '/pollerwebservice/TopologyService.asmx/DeleteDevice?deviceNameOrIpAddress=', a.dev_name, '\'><img border=0 src=\'images/deleteX.gif\' /></a>' ) as deletethis from service_managers b, devices a inner join ( select dev_alias as dupon from devices where model_id<>0 group by dev_alias having (count(dev_alias)>1) ) dup on a.dev_alias = dup.dupon where b.smgr_id = a.dev_owner order by a.dev_alias; ]]> </SelectCommand> <odbcconnectionstring Name="NetVoyant">Driver={MySQL ODBC 5.1 Driver};Server=192.168.20.21;Port=3308;Option=3;Database=nms2;User=nmsuper;Password=redpoint;</OdbcConnectionString>
To create the view, run the following SQL commands against the NPC server:

INSERT INTO controls SELECT if(min(id)-1>0,min(id)-1,29999), 'NetVoyant Duplicates', 'Graph', 'NetVoyant-ODBC', '/nqWidgets/Integration/wicTable.ascx', 1, 0x200000, 0, 0, 0, 1, '|nc|g|', '', 'Y' FROM controls WHERE datasourcetype=0x200000; INSERT INTO control_properties (ControlID, Editable, PropertyName, PropertyValue) VALUES (-1, 'N', 'Title', 'NetVoyant Duplicates'), (-1, 'N', 'DisplayColumns', 'deletethis;Alias;ExpiringSince;UnresponsiveSince;Poller'), (-1, 'N', 'DisplayFormats', 'string,string,string,string,string'), (-1, 'N', 'DisplayNames', 'Action,Device Alias,Expiring Since,Unresponsive Since, Poller'), (-1, 'N', 'ColumnWidths', ',,,,'), (-1, 'N', 'HorizontalAlignments', 'Center,Left,Left,Left,Left'), (-1, 'N', 'SortField', 'aliassort'), (-1, 'N', 'SortDirection', 'Desc'), (-1, 'N', 'Integration.Parameter.SelectCommandName', 'NVDuplicates'), (-1, 'N', 'Integration.DataType', 'SelectCommand'), (-1, 'N', 'Integration.Parameter.DateTimeFormat', 'yyyy-MM-dd HH:mm'), (-1, 'N', 'Integration.Parameter.TopN', '200'), (-1, 'N', 'Integration.Parameter.OdbcConnectionStringName','NetVoyant'); UPDATE control_properties SET ControlID = ( select min(id) from controls where datasourcetype=0x200000) WHERE ControlID = -1;

NPC and the footer.text Property

Often when displaying report data, it is handy to have a little explanation about the data being presented.  In the case of NPC, this is accomplished by means of the footer.text property.  As it turns out, this footer.text is HTML based and can use some fairly advanced HTML features.

I say this can be done in NPC, but it can't; at least unless you jailbreak NPC.  At this point, CA has decided that CAPC will replace NPC.  Currently, there's no upgrade path and many of the features that some customers rely upon haven't been rebuilt into CAPC (remember CAPC is a rebuild from the ground up, so just because it was in NPC doesn't mean it will be in CAPC). I've already posted on the community about some of the features that I require before I will upgrade myself or anyone else to CAPC (browser view for example).

However, since NPC now has one foot in the grave (CA is evangelizing IM2.0 which requires CAPC more than Jim Jones' kool-aid), I see no reason why anyone wouldn't want to start experimenting with some of the hidden features in NPC.  So, if you haven't jailbroken NPC yet, you might consider it.  However, if you don't have NV as a data source, don't worry about it.  The NV view wizard is all you really get.  While that opens tons of doors to customization you never thought possible, it doesn't do anything for non-NV customers.  If you're interested, email me.  Once CA puts the last nail in NPC's coffin, I'll publish the steps here.  Until then, I need deniability.

So, on to the topic of this post.  When using the view wizard to create/edit NV views in NV (or NPC if you've jailbroken it) it turns out you can put some fairly complex html in the footer field.  Obviously, you can use {Resolution} and {Samples} in there to pull the current resolution and data point count (whether it's sampled data or not).  However just today I tried and was able to put a fully formed html table in the footer. I then tried to take it a step further and embed a YouTube video.  That worked fine as well.

In case you don't see it, this can be useful when trying to explain to new users what certain views mean.  For example, I could put together a complete page with live data and YouTube videos or text explaining what each one means and how to use it.  For example, I could put some text or an image in the footer that could expand a DIV section containing a YouTube video explaining how to interpret the data.  The nice part about this is that the footer is tied to the view.  Previously, I would put a separate browser view embedding the help information.  This is good for situations where you want to put the data on the left and explanation on the right.  However, putting the explanation in the footer ensures that the explanation stays with the data even through moves and copies.

So in order to put a hidden YouTube video in the footer of a view:

1. First check to make sure the view supports a footer (most do, this is just a sanity check).
2. Install HeidiSQL and connect to your NPC server.
3. Go to the control_properties table and filter it to only show your view.  Get the controlid from the NPC web gui by looking at the status bar when you open the view menu and mouseover the edit option.  Then set a filter for controlid=X, pageid in (0, Y), and propertiesid in (0, Z).
4. Find the footer.text property.  
5. Edit the propertyvalue column and insert your html.  
6. Save the record and you're good to go.

If you change the footer.text where pageid=0, propertiesid=0, and userid=0, you are modifying the default definition of the view and any existing views tied to the default settings and any new views will have this modified setting.

Here's the code you would insert if you wanted to embed my SA video.  You'll need to take out the carriage returns to make it one long line:

<script language="javascript"> function toggle(showHideDiv, helpimg){ var ele=document.getElementById(showHideDiv); var img=document.getElementById(helpimg); if(ele.style.display=="block"){ ele.style.display="none"; img.src='../npc/images/expand.gif' }else{ ele.style.display="block"; img.src='../npc/images/contract.gif' } }</script> <a href="javascript:toggle('hiddendiv','helpicon');"><img border="0" id="helpicon" src="../npc/images/expand.gif" /> Show help</a> <div id="hiddendiv" style="display: none;"> <iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="http://www.youtube.com/embed/9vPZRViW43c" width="420"></iframe></div>

Creating Properties for sysName, sysDescr, and sysObjectID

UPDATE: I've combined this tool with the tool I built that allows administrators to add/delete/rediscover devices without logging into the console.  This tool is combined with the properties creator simply because they both need to run right after discovery.  It actually has 3 parts: a widget, a JavaScript file and the batch file that runs every night.  First the batch file, which is an expansion of the properties creator.  In addition to the normal task of adding properties for new devices, this also adds a pair of properties that, when rendered on the NPC device details page, present the administrator with a rediscover button and a delete button.  By default, a password is required.  The password is set in the external JavaScript file (below) on lines 2 & 16 for rediscovery and deletion, respectively:


Next is the JavaScript file, which must be in the custom virtual directory (with alias 'custom'):


Lastly the widget.  The widget is only for adding new devices.


If you want to add a delete button, remove the text 'style="display:none;"' from the widget source.

---

Occasionally, I find it necessary to build auto-enable rules in NetVoyant based on SNMP properties like sysName, sysDescr, and sysObjectID.  Unfortunately, these are not all available for every dataset as SNMP parameters that could be used in rules.  However, custom properties are always available in auto-enable rules (not discovery rules since property comparison happens after initial discovery).  What this means is that rules can be built to automatically disable poll instances according to model, OS version, or software version (as obtained via the sysDescr).
In order for this to work however, each device needs custom properties.  Setting these manually is a pain and would take forever for anything other than a lab system.  To combat this, I've built this script (thanks Wade for the query help) that creates custom properties for every SNMP capable device containing the sysDescr, sysObjectID, sysName, sysContact, and sysLocation.

@echo off
set sqlcommand=mysql nms2 --skip-column-names -e "select count(*) from devices where snmp_capable=2 and dev_properties
set propertieslist=(select property_set_id from properties where property_name=
set logfile=D:\updateproperties.log
echo %date% - %time% - Script Started >> %logfile%
for %%A in (sysDescr,sysObjectID,sysName,sysContact,sysLocation) do (
 echo Devices with %%A property: >> %logfile%
 %sqlcommand% in %propertieslist%'%%A')" >> %logfile%
 echo Devices without %%A property: >> %logfile%
 %sqlcommand% not in %propertieslist%'%%A')" >> %logfile%
)
echo Running query  >> %logfile%
set inspropsql=mysql nms2 -e "replace into properties (select dev_properties,
set inspropsql2=0, 0 from devices where snmp_capable=2)"
%inspropsql% 'sysDescr', 18, sys_descr, %inspropsql2%
%inspropsql% 'sysObjectID',18, sys_objectid, %inspropsql2%
%inspropsql% 'sysName', 18, sys_name, %inspropsql2%
%inspropsql% 'sysContact', 18, sys_contact, %inspropsql2%
%inspropsql% 'sysLocation', 18,sys_location, %inspropsql2%
for %%A in (sysDescr,sysObjectID,sysName,sysContact,sysLocation) do (

 echo Devices with %%A property: >> %logfile%
 %sqlcommand% in %propertieslist%'%%A')" >> %logfile%
 echo Devices without %%A property: >> %logfile%
 %sqlcommand% not in %propertieslist%'%%A')" >> %logfile%
)
echo %date% - %time% - Script Ended ----------------------------------->> %logfile%

This script has to be run on the poller(s).  A new device will not get the properties until the script is run again, so, it is probably best to set it to run as a scheduled task every night right after discovery.

Once you've got the properties in place, you can create auto-enable rules using these properties by referencing them with the $.  So, for example, if I wanted to disable ifstats monitoring on all devices that have a sysLocation like 'France', I would add the following to the Property Rule in the add auto-enable rule dialog box:
$sysLocation like '%France%'
Save the rule, apply it to the dataset, then rediscover the device.  Voilá!

GXMLG 2.0

I finally broke down and rebuilt my GXMLG tool.  Given the complexity of the task, I originally wrote the applet using MS Access.  However, to make things easier to distribute and easier to troubleshoot and use, version 2.0 uses perl and is run from the command line.  To illustrate the difference, the old utility was 6.5MB.  The new script is 11KB.  That's what a graphical interface gives you.



You'll have to install perl (I use strawberry perl on windows boxes) and run the script like this:

>perl gxmlg.pl

Running it without any arguments shows you the help file:

This script outputs any combination of configuration files for the NetQoS
suite of products. You must install strawberry perl and Text::CSV,
Text::CSV_XS, and Getopt::Long. To install CPAN modules, run cpan [module name]
from the command prompt.

Example: cpan Text::CSV

This script was created by Stuart Weenig (C)2012.  For more information visit
http://stuart.weenig.com. This script may be redistributed as long as all the
files are kept in their original state.


Current Version: 2.0

Usage: PERL gxmlg.pl [-outnpcxml] [-outsacsv] [-outnvcsvnv] [-outucmcsv]
                     [-infile NAMEOFINFILE] [-npcinspath INSERTPATH]
                     [-npcxmlname NPCXMLFILE] [-sacsvname SACSVFILE]
                     [-nvcsvname NVCSVFILE] [-ucmcsvname UCMCSVFILE]

    -infile NAMEOFINFILE        Specifies the name of the Sites file to be
                                imported. (If omitted: sites.csv)
    -outnpcxml                  Output an NPC XML groups definition file.
    -npcxmlname NPCXMLFILE      Name of the NPC file to be output. (If
                                omitted: NPCGroups.xml)
    -npcinspath INSERTPATH      The path to the group that will serve as
                                the insertion point.  Required if using
                                -outnpcxml option.
    -outsacsv                   Output a SA networks CSV file.
    -sacsvname SACSVFILE        Name of the SA file to be output. (If omitted:
                                SANetworks.csv)
    -outnvcsv                   Output a NV discovery scopes file.
    -nvcsvname NVCSVFILE        Name of the NV file to be output. (If omitted:
                                NVScopes.csv)
    -outucmcsv                  Output a UCMonitor locations file.
    -ucmcsvname UCMCSVFILE      Name of the UCMonitor locations file to be output.
                                (If omitted: UCMLocations.csv)

You must install strawberry perl and Text::CSV, Text::CSV_XS, and Getopt::Long.
To install CPAN modules, run cpan [module name] from the command prompt
Example: cpan Text::CSV

If the server you will be running this on doesn't have access to the internet, you
won't be able to install the modules automatically (since they have to be downloaded
from the internet).  The solution is to download the tarballs from www.cpan.org and
extract them using winzip or winrar or 7z.  You might need to extract several times
until you get just the folder with the files in them.  Then copy them to your server.
Open a command prompt and cd to the directory containing Makefile.pl (you'll have to
do this for each module).  Then execute the following:

     perl Makefile.pl && dmake && dmake test && dmake install

For the text modules, do Text::CSV first, then Text::CSV_XS.

The script itself is pretty simple.  Specify an input file.  This input file is the same sites/networks file referenced in my previous blog post.  Here's a sample sites list to get you started.  Then just decide which output files you want.  If you specify the npc output file, you'll also need to specify the insertion point (more information about the insertion point).

If you don't have internet access on the box, you won't be able to install the Text::CSV modules to install (since they come from the internet).  The solution is to download the Text::CSV and Text::CSV_XS tarballs and extract them using winzip or winrar or 7z.  You might need to extract several times until you get just the folder with the files in them.  Then copy them to the NVMC.  Open a command prompt and cd to the directory containing Makefile.pl (you'll have to do this for each one).  Then execute the following:

perl Makefile.PL && dmake && dmake test && dmake install

Do Text::CSV first, then Text::CSV_XS.

NetVoyant Rollups: Sums, Maximum, Percentiles, etc.

For most situations out there, the default rollup is perfectly fine.  What i mean is that when you add an expression to a dataset, the default rollup (which is an average) is exactly what someone would be looking for in a rollup.  If i show top interfaces for an hour, I'd like to sort those interfaces by the highest average utilization, which means i want NV to take an average of the utilization data points during that hour.

However, in some situations, it may be more accurate to calculate a different rollup.  For example, if i wanted to, i could have NV calculate both the average value of all the data points collected in the last hour and also calculate the standard deviation so that i know how consistent my utilization is.  Higher standard deviation means there are at least some points that are far away from the average.  I could also have NV calculate the maximum or a percentile of all the points from the last hour.  By adding max and percentile to a view, i can easily see more clearly what is happening on an interface.

One other situation is volume.  If you're polling some OID for some kind of volume (KB or MB), the first thing you should do in your expression is put it in bytes.  This allows you to take advantage of the auto scaling feature in the views.  This means that instead of showing numbers like 12000000 along the dependent axis, NV can display something like 12.  You'd then put {Scale} in the axis label so that KB, MB, GB, etc. is displayed indicating the unit.
The next thing you'd do for volume is change the rollup.  Obviously if you're tracking volume, having an average of all the points collected in the last hour is useless.  What you really want is a sum of the volume in the last hour.  To do this, remove all rollup types.

Did i mention how to do that?  I guess i didn't.  Edit the expression and click the advanced button.  Uncheck all the checkboxes so that the rollup is a sum instead of an average.

Another trick about rates:
If you're polling an OID and want to convert it to rate, create a new expression and divide the expression by the variable 'duration'.  Duration is always equal to the number of seconds in a poll cycle.  Technically it's the number of seconds since the last poll, so you do have to be a little careful about that.
Again, if your OID is in some unit like KB, convert it to bits (KB*1024*8).  Then when you divide by duration, you get bits per second.  By setting the view auto-scale to rate, NV will automatically convert it to the needed value (Kbps, Mbps, Gbps, etc.).

NetVoyant Duplicates Finder

UPDATE: This method has been replaced by the ODBC method.

---

I've been working for a while now on a good way to find and remove duplicates from NetVoyant.  Luckily, there is a web service that can delete devices (more on NetQoS web services).  All you need is the device IP address and the poller (to build the web service URL).  I played around for a while trying to build something in a windows batch file and couldn't get it to do what I wanted to do.  So, I reverted to Perl (which I probably should have done from the beginning).  Anyway, the result is a script that can be run regularly on the NetVoyant master console.  The output is a CSV file and an html file.  The CSV file contains the output from the brains of the duplicate finder script, namely: a list of every device that exists more than once in the NV system, along with the device properties including the poller.  The CSV file is output to the script directory.  The script can be configured to output the html file wherever you want.

After that, the script uses perl to wrap the data in the CSV into an html widget.  The widget shows the same data as the CSV as well as a link on every line to delete the device.  As long as the NV pollers resolve by name, the link should work to delete the device and its corresponding scope.  If you only want the CSV, edit the batch file and comment out the call to the Perl script (i.e. put 'rem' in front of the line that start with the word 'perl').

If you do want the HTML, you'll need to install Strawberry Perl and download a couple of modules.  Installing Strawberry Perl on the NetQoS boxes isn't a new thing.  Most of the developers and support guys have Perl installed on their test boxes and I've had it installed on many customers' boxes.  The install doesn't require a reboot and you can take all the defaults.  After doing the install, open a command prompt and type the following:

D:\>cpan Text::CSV
D:\>cpan Text::CSV_XS

Perl will download and install the necessary modules and return you to the command prompt when its done.

After that, all you need to do is download the zip and extract the files to somewhere on your NVMC.  Setup a scheduled task to run the batch file every so often.  The web page doesn't update unless the script runs (it doesn't refresh the list of duplicate devices simply by refreshing the page).

To get the script to output the html file to somewhere other than the script directory, go to the makehtml.pl file and modify the line that starts with 'my $outputfile = ' and update the output file path and name.  For example:

my $outputfile = 'D:\\NetVoyant\\Portal\\WebSite\\dupslist.html'

Perl requires a double backslash since a single backslash is the escape character.

That's it.  You're done.  You can use the browser view to put the resulting html file on an NPC page if you've designated a destination that is served up by the NVMC's IIS web service.

Enjoy!  If you have improvements, please let me know so I can update the source.

P.S. If you don't have internet access on the box, you won't be able to install the Text::CSV modules to install (since they come from the internet).  The solution is to download the Text::CSV and Text::CSV_XS tarballs and extract them using winzip or winrar or 7z.  You might need to extract several times until you get just the folder with the files in them.  Then copy them to the NVMC.  Open a command prompt and cd to the directory containing Makefile.pl (you'll have to do this for each one).  Then execute the following:

perl Makefile.PL && dmake && dmake test && dmake install

Do Text::CSV first, then Text::CSV_XS.

NPC and NetVoyant Web Services Gadgets

In combination with my method of inserting custom content into NPC, I've created a couple of gadgets that can be added to NPC.  These gadgets give the viewer access to the NPC and NetVoyant web services that can be used to import and export group definitions, add devices to NetVoyant for polling, delete devices from NetVoyant, and delete discovery scopes from NetVoyant.

Installation Instructions

• You can download them here [link removed, see the Tools page].
• Extract the zip contents to the custom content directory described here.  
• Edit the html files to update the URL.
• Look for the <form> tag and update the action string to point to your NPC or NV server (depending on the html file).
• Put three browser views on a page that only NPC administrators can access.  
• Edit the three browser views to point to the following URLs (you should probably update the view names as well and I like to hide the view border)
• NPCGroupExport.html
• NPCGroupImport.html
• NVDeviceMgmt.html [this widget has been moved here]

You should now have a page showing all the controls.  Here's what they do:

NPCGroupExport.html

This gadget exports XML for any group in NPC.  This XML describes the group structure and defines any rules for the groups.  This can be used in conjunction with NPCGroupImport.html to modify or populate NPC groups programmatically.

NPCGroupImport.html

This gadget imports XML into NPC to redefine groups.  This can be used to redefine groups, rules, or membership.

NVDeviceMgmt.html [this widget has been moved here]

This gadget facilitates adding and removing devices from NetVoyant.

I hope you enjoy them.  If you have any improvements, please let me know so I can update the source.

SNMP Counter32 vs. Gauge32

I ran into a problem recently where a manufacturer had built a MIB that contained an OID for 'objects in cache' with syntax Counter32.  However, when polling the value of that OID, it was discovered that the OID didn't behave like a Counter32; it went up and down (Counter32 is supposed to only go up; a lower value than the previous poll indicates a roll-over).  It occurred to me that the manufacturer probably meant to indicate the current number of objects in cache and mistakenly set the syntax to Counter32.  Since the actual number of objects in cache can rise or fall, a Counter32 wouldn't accurately represent this.  Instead, a Counter32 would indicate how many items had been added/removed from the cache since the previous poll (since most NMS systems would take the delta between the previous counter value and the current counter value).  While knowing how many items had been added or removed from the cache since the previous poll might be useful, what is probably more useful would be the total count.  The difference actually has nothing to do with the value returned by the device.  The problem is that since the MIB indicated that the OID is a Counter32, most NMS systems interpret that type of object differently, performing a delta instead of reporting the actual number returned by the device.

The fix for this is to change the way the NMS system interprets the returned value by changing the MIB.  In this case, the syntax needs to be changed from Counter32 to Gauge32.

Here is what the MIB contained originally:

 proxyNumObjects OBJECT-TYPE
  SYNTAX Counter32
  MAX-ACCESS read-only
  STATUS current
  DESCRIPTION
   "The number of objects currently held by the proxy."
 ::= { proxySysPerf 2 }

Here is what the MIB needs to be changed to:

 proxyNumObjects OBJECT-TYPE
  SYNTAX Gauge32
  MAX-ACCESS read-only
  STATUS current
  DESCRIPTION
   "The number of objects currently held by the proxy."
 ::= { proxySysPerf 2 }

Changing this syntax in the MIB and recompiling into the NMS system instructs the NMS to use the raw value returned instead of performing a delta with the previously obtained value.

In NetVoyant, recompiling the newly edited MIB will be sufficient to correct this problem.  However, a restart of the Mibs service is required before the newly compiled syntax gets used.  Since everything depends on the Mibs service, everything will get restarted.

Managing NetVoyant through Web Services

Web services can be used to manage NetVoyant devices.  I've had at least one customer who built auto-provisioning of NV monitoring through web services.  It can be a headache, but it is possible.  I won't go into details about how to actually automate the use of the web services; go take a class for that.  This practice actually goes against my preferred method of adding all devices to NV and using class designations, auto-discovery rules, and auto-enable rules to manage what gets monitored.  However, in some cases, that can't be done.  Here are the details on the web services.

In any distributed system, there are two types of systems: NV Master Console and NV Poller.  NV uses scopes to manage the devices to be monitored.  Each scope details an IP address to be monitored, a subnet mask, and the poller responsible for monitoring it.  The process of adding and removing devices from NV involves manipulating the scope that corresponds to the device to be added or removed.

Adding a device to NetVoyant

If you are using web services to add devices to a distributed NetVoyant system, you'll either want to add the device to the poller responsible for polling all the devices in that region OR you'll want to add it to the poller that is least loaded (in terms of total number of devices).  It's also possible that you may have multiple pollers covering a single region and need to determine which poller is least loaded.In order to find out the number of devices on each poller, the GetDeviceCount operation of the NetVoyantService should be invoked on each poller (and not the master console).
The NetVoyantService can be found at /pollerwebservice/NetVoyantService.asmx?WSDL on each poller.  By invoking this procedure without any arguments, the result will be XML indicating how many devices total are on that particular poller.  Unfortunately, this must be run against each poller to get that poller's count.  Below is shown a sample output.

<?xml version="1.0" encoding="utf-8" ?>
<int xmlns="http://netqos.com/NetVoyantWS/">41</int>

How to Use Poll Instance Properties in a View

If you aren't completely, 100% comfortable with manipulating the database manually, stop reading this post now.  If you continue reading this post, you do so at your own risk.

In NV (and NPC if you've jailbroken it), you can create views using the NV Custom View Wizard.  One of the things the developers did but didn't really publicize is that you can pull poll instance properties into those views.  This includes any custom properties you may have added to the poll instance.  By default all poll instances have at least two properties: 'Name' and 'Description'.  Built in dataset poll instances (like ifstats) will have other properties which you can also use.  The trick is that you have to manipulate the database slightly in order to instruct the view to even look at the properties.  It isn't too bad, but like I said before, if you don't know how to do this with your eyes closed, don't try it.

1. First you have to get the control id of the view (technically called a control) you have just created and will be modifying.  This is pretty easy to get, just open the view in the wizard and look in the url for the controlid parameter.  In this example, the id is 1200031.
2. Next you need to get into the database and take a look at the control properties for your control.  Something like this: 'select * from control_properties where controlid=1200031 order by pageid, propertiesid, userid, propertyname;'

That should result in something like the following.

ControlID	PageID	PropertiesID	UserID	PropertyName	PropertyType	PropertyValue	Editable	Enabled
1200031	0	0	0	ColumnNames	string	latencymin,latencyaverage,latencymax	Y	Y
1200031	0	0	0	ControlType	string	PollInstance	Y	Y
1200031	0	0	0	Create.Date	string	Added at 1/13/2012 9:05:07 AM by nqadmin	Y	Y
1200031	0	0	0	data.chartType	string	Table	Y	Y
1200031	0	0	0	DataSetName	string	ISILONPerf	Y	Y
1200031	0	0	0	Description	string	This report focuses on the worst values for the specified parameter and therefore may be more prone to problems or failure.	Y	Y
1200031	0	0	0	DisplayFormats	string	ms|ms|ms	Y	Y
1200031	0	0	0	DisplayNames	string	Minimum,Average,Maximum	Y	Y
1200031	0	0	0	drillDown.target	string		Y	Y
1200031	0	0	0	FieldNames	string	latencymin;latencyaverage;latencymax	Y	Y
1200031	0	0	0	FieldNames2	string		Y	Y
1200031	0	0	0	FieldNames3	string		Y	Y
1200031	0	0	0	FieldNames4	string		Y	Y
1200031	0	0	0	FieldNames5	string		Y	Y
1200031	0	0	0	Filepath	string	/nqWidgets/Poller/wptTopN.ascx	Y	Y
1200031	0	0	0	footer.text	string		Y	Y
1200031	0	0	0	Limit	string	10	Y	Y
1200031	0	0	0	MibTables	string	ISILON_MIB.nodeProtocolPerfEntry	Y	Y
1200031	0	0	0	OrderBy	string	latencyaverage DESC	Y	Y
1200031	0	0	0	PropertyNames	string	Description,Name	Y	Y
1200031	0	0	0	RedThreshold	string		Y	Y
1200031	0	0	0	Title	string	Isilon Node Performance Table by Protocol	Y	Y
1200031	0	0	0	Wizard.Action	string	ReportWizard('/npc/ReportWizard.aspx?PageID={PageID}&CtrlID={CtrlID}&PropertiesID={PropertiesID}');	Y	Y
1200031	0	0	0	yaxis2.ColumnNames	string		Y	Y
1200031	0	0	0	yaxis2.DisplayColors	string		Y	Y
1200031	0	0	0	yaxis2.DisplayFormats	string		Y	Y
1200031	0	0	0	yaxis2.DisplayNames	string		Y	Y
1200031	0	0	0	YellowThreshold	string		Y	Y

3. You'll notice that I have a property called PropertyNames.  This is the entry you need to add to this table.  The propertyname will be 'PropertyNames' as shown.  The propertyvalue will be a comma separated list of the properties you would like to be available to this view.  For example, since I want both the description and the name properties available to this view, I added them both in there.
   You would normally add this record using an insert SQL statement.  A 0 in the pageid, propertyid, and/or userid fields represents a wildcard.  Technically, a view can exist multiple times on a page.  If it does, the pageid will contain the id of the page on which the control instance(s) exist(s).  The propertyid refers to which instance of the view on the page.  The userid refers to any properties that the user may have modified and saved to only their account.  If you want this change to apply to all instances of the view, the easiest would be to make the change to the default (pageid=0, propertyid=0, userid=0) then re-customize any views that have been customized.
4. Once you've got that done, you can go back to the view wizard.  Go to the 4th step.  You can now use the property values within expressions or the 'Where' field.  The syntax is as follows: p_PropertyName.property_value, where PropertyName is the name from the comma separated list you entered into the database.  In my case, if I wanted to display the description, I would create an expression called 'Description' and the expression formula would be 'p_Description.property_value'.

One of the nice things about exposing properties is that you can use the properties in the 'Where' field of the view.  The where field allows you to pre-filter the view so that only certain objects show up.  For example, if I knew that the description contained an enumerated set of values (cifs, nfs, http, other), I could create a view that only shows the cifs objects by putting "p_Description.property_value = 'cifs'" in the Where field.

Unfortunately, this doesn't work for device properties like sysLocation, sysContact, sysName, sysObject, etc.  However, there is a script floating around that will take those parameters and store them in custom poll instance properties, which could then be exposed using this method.