UCDavis MIB for Monitoring Linux Memory

I discovered quite some time ago that the Net-SNMP agent on my RaspberryPi doesn't report memory utilization in the hrstorage MIB like I would expect it to.  It's not wrong, however, the value doesn't match up with what's actually being used on the device.  The reason for that is that Linux can use the system RAM for several things: processes, shared memory for processes, buffers, and disk cache.  When most people ask how much memory is in use, they are asking for how much memory is in use by the processes.  That's the value you get on the second row under the 'used' column of the output of the free command:
total used free shared buffers cached Mem: 497504 486784 10720 0 55596 397996 -/+ buffers/cache: 33192 464312 Swap: 102396 0 102396
However, the OIDs in the hrstorage MIB actually return the value from the first row of the 'used' column.  The problem is that both of these numbers represent memory utilization.  The first row shows the total of the processes, shared, buffers, and cache.  That's the total amount of memory that's in use on the system.  However, this isn't the value most people associate with memory utilization.

In order to get the correct value, there are two options.  The first doesn't work with NetVoyant, but it doesn't use additional MIBs or OIDs to get the data.

Since the shared, buffer, and cache memory is reported in the hrstorage table, you can simply take hrStorageSize of the Physical Memory row (hrStorageType==1.3.6.1.2.1.25.2.2), and subtract the hrStorageUsed from the shared, buffer, and cache rows (hrStorageType==1.3.6.1.2.1.25.2.1.1).  Since NetVoyant can't use values from other poll instances in an expression, it won't work in NV.

Side Note: This may be possible by creating a single expression that results in the positive value of hrStorageSize when the hrStorageType is .2 and a negative value of hrStorageUsed when hrStorageType is .1.  The sum of that expression for all the .1 and .2 poll instances should give you the total used memory.  However, since the sum could only be done in a view in the web GUI, it would only work for reporting and not thresholding/alarming.

The second option is to use the UCDavis MIB.  The NetSNMP agent does populate the UCDavis tables, so any of the values there can be polled.  The problem is that there's no real clear documentation on which OIDs give you which values when compared to the output of the free command.  Here's the mappping:
total used free shared buffers cached Mem: A B C D E F -/+ buffers/cache: G H Swap: I J K
Given the output above, here are the OIDs or combinations you need to calculate the values:

1. memTotalReal
2. memTotalReal - memAvailReal
3. memAvailReal
4. memShared
5. memBuffer
6. memCached
7. memTotalReal - memAvailReal - memShared - memBuffer - memCached
8. memAvailReal + memShared + memBuffer + memCached
9. memTotalSwap
10. memSwapError
11. memAvailSwap

Given this, it should be pretty easy to create a dataset to poll memory.  Just remember, these OIDs are in units of KB, so if you want it in Bytes so that NV automatically scales (to KB, MB, GB, TB, etc.) you'll need to multiply each one by 1024.  Obviously, if you're calculating % utilization, you don't need to multiply both the numerator and the denominator by 1024.  You will need to multiply by 100 to get the ratio to a scale of 0-100%.

Combining All Files into a Single Batch

One thing I figured out how to do was to simplify the installation of a complex set of files that are used as auxiliary files for a master batch file.  For example, for the tool that I built over the last few weeks, I ended up with gatherer.vbs, entitycleaner.pl, and tsv2csv.pl that were all called from within the master batch file.  While I was developing things, it was handy to have these as individual files because I could easily edit them individually.  However, when installing the tool on the final server where it needed to go, I had to zip up all the files (including some other files that I haven't released yet), copy them over, and unzip them (replacing the existing ones if they were already there).  This wasn't too efficient and I needed an easier way to transmit the scripts.

The solution I came up with was to embed the contents of the auxiliary files into the master batch file.  I then instructed the master batch file to echo the auxiliary file contents into the files at runtime.  This meant that I only had to transmit the single batch file to the target server and run it.  When it ran, it would create the files it needed, use them, then destroy them.  This also kept the working directory clean.

There were a few caveats, however.  The biggest hurdle is trying to echo the special characters that other programming languages use from within a batch file.  There were two main resources that I used to make sure things worked right.  The first is Rob VanDerWoude's web page on escaping special characters.  This helped me determine which characters from the source script would have to be escaped when echoing out from the batch file to the auxiliary file.  The second utility I used was http://text-compare.com/.  It allows you to compare one text file to another and tells you where the differences are.

The next thing I did was create a simple batch file to echo each line of the content out to a new file.  I then took the original content and compared it to the content created by the simple batch file.  For example, for tsv2csv.pl on the left, I created the batch file on the right:

You can see that the batch file on the right uses the carat (^) to escape several of the characters.  Since I used one redirector at the end (line 11) to output the entire thing into the test output file, lines 2-10 needed to be in parenthesis.  This means that all parentheses needed to be escaped.  Also, since > and < are redirectors in Windows Command, they also had to be escaped.  Interestingly enough, since the > on line 8 is in double quotes, it doesn't need to be escaped.  This is only possible since I needed the double quotes anyway in the auxiliary file.  If I had used single quotes on line 8, I would have had to escape the >.

In this way, I combined all the auxiliary files into one single batch file.  I added the lines from the test batch file into a function in the master and called the function from the main program section of the master batch.

Some other characters that have to be escaped are the ampersand (&), pipe (|), and percent (%).  The & and | were easily escaped.  However % proved to be more difficult.  Normally, the % sign is followed by a non-alphanumeric character (space or punctuation).  However, when trying to output a date format for SQL, the % sign is followed by characters (i.e. %Y-%m-%d).  This causes a problem because Windows command is reads each of those a variables.  You can't even escape them because in a batch file, the double percent (%%Y) reads as a variable too.  In the end, what I had to do was store 'Y' in a variable called 'FormatY' like this:

set formatY=Y
echo %%%formatY%

The first two percent signs read as an escaped percent sign and a single % is echoed.  The %formatY% resolves to Y and a single Y is output.  To get the whole string, I did this:

set formatY=Y
set formatm=m
set formatd=d
set formatH=H
set formati=i
set formats=s
echo %%%formatY%-%%%formatm%-%%%formatd% %%%formatH%:%%%formati%:%%%formats%

The output looks like this:

%Y-%m-%d %H:%i:%s

This is a messy way of doing things, but it works.

Convert Tab Separated Values to Comma Separated Values

Using MySQL to output CSV files can be a bit of a pain.  You have two options depending on where you want the file to end up.  If you want to write the file to the server, you can append your query with a bunch of lines detailing what separator character to use, what file name to output to, and what text delimiter to use.  This is great if you want the file to be on the server.  However, if you're accessing MySQL remotely, you usually want the file to be saved locally.  This is fairly easy, but the simple method only outputs in a tab separated values file instead of comma separated.  Since I wanted CSV, I designed a short Perl script to go through a TSV and change all the tabs to commas.  This is very similar to the entities cleaner posted earlier.



Line 2 opens the file specified in the first argument.  Line 3 sets up the variables.  Lines 4-9 replace the \t character with a comma.  Lines 10-11 close the input file and open the same file as the output file (using clobber to overwrite the original contents).  Line 12 outputs the cleaned lines to the output file and line 13 closes the file.

Entity Cleaner

The entity cleaner was a workaround for a problem I didn't want to take more time to resolve.  I have a script that queries a MySQL database and outputs an html table.  This works great, however, I wanted to be able to format some of the content in the table, in this case, align the text to the right side of the cell.  The code to right align the content was easy.  However, when MySQL outputted the HTML, the HTML I had added was converted into HTML entites.  So, instead of showing the content right aligned, it showed all the markup around the content.  So, I needed to do some post processing of the file to convert the HTML entities to their actual characters.  This Perl script was the answer.  Since my output only contained three entites, they are the only ones I replace here.  However, this could be easily expanded to include all the basic entities.

Line 2 gets the input file name and stores it for later use.  Line 3 sets up the array that will contain the content as it's being cleaned.  Line 4 sets up a counter.  Line 6 opens the file.  Lines7-14 clean the entities.  Line 8 grabs the next single line from the input file.  Line 9 stores that line in the corresponding element in the array.  Line 10 replaces the &lt; with <.  Line 11 replaces the &gt; with >.  Line 12 replaces &quot; with ".  Line 13 moves us to the next line.

Lines 15-18 closes the intput file, sets the output file to the input file (change in place) and opens the file for outputting.

Lines 19-23 output the contents of the array to the output file and give a confirmation message.

Gatherer

One of the major pieces of the tool that I spent the last few weeks working on is the gatherer.  It's written in Visual Basic since it was the easiest thing to use without installing additional modules (like I would have had to do with Perl).  The purpose of this script is to download a file via http and save it locally.  I used a modified version of this final script in my tool.

The script must be called with 2 arguments.  The first is the name of the server to fetch from.  The second is the file to fetch.  This is a simplified version of my script.  It could be simplified further to just provide a URL instead of the two pieces.  If you want that, tell me in the comments and I'll modify it and post it.

Line 7 takes the first argument and stores it in the targetServer variable.  Lines 8-9 build the final URL to fetch.  This is where the simplification could be made to just fetch a provided URL.

Lines 10-13 actually fetch the file and echo back the response code.  Lines 14-16 check that response code for any errors and exit the script with an ERRORLEVEL of 1 if there was a problem.  The only thing this doesn't check for is 'Server not found' errors.  Maybe I'll look into that problem later.

Lines 17-32 execute if the HTTP response code doesn't indicate a problem.  Lines 18-24 get the current working directory so that the downloaded file can be saved there.  Lines 25-31 actually save the file.  The file is saved with a modified file name in the format [servername][originalfilename].

Batch File Template

I've done some major work over the last few weeks with batch files.  In fact, I just finished a tool that utilizes 5 different languages (Windows batch, Perl, VBScript, HTML, & SQL) to accomplish some pretty cool stuff.  In the process of building the tool, I had to set some standards in the way I work with the various programming languages.
One major step was to finalize (at least in my mind) my method of writing batch files.  So, I came up with the following template that I'll probably use for any projects going forward.  I'll explain the various features:



The first section of code (lines 2-7) sets the default variables that this script will use.  This section can be used to declare the default value for any number of variables.  It also sets the current version and last updated date.

The next section (lines 8-19) branches out to the built in help.  This will not execute the main program and instead will go to the help section (lines 42-51) and execute the code there.  Most of the time, I just put echo statements there to echo the help documentation out to the screen.  The GOTO:EOF simply returns execution control to the calling process.  For help this means returning to the command prompt.  If some arguments are required, you can easily branch to the help section by unREMarking line 9.  This will execute the help commands and exit if no arguments are specified.  This is handy because you can then just run the batch file without any arguments and see the help (the way most built in commands work).

The next section (lines 20-26) reads in the command line arguments.  If the switches for variable1 and variable2 are present, the value immediately after the switch is read into the variable (for numbers and strings).  If the switch for variable3 is present, variable3 is set to 1 (boolean).  The shift and GOTO Loop commands cycle through all the arguments until none remain (line 21).

The next section (lines 27-35) is the main section of the program.  The first thing that happens is that the version is echoed by calling the version function (lines 52-54).  In this case, :version acts like a function (with no returning value).  This is handy since version can be called multiple times in different places in the main program execution (for example outputting the version at the end of any output files).  The version function can be extended to include author information or links to online documentation.

The next section (lines 39-41) is a custom function that is called from the main program.  It is called on line 34 with one argument.  Normally arguments separated by a space are treated as two, but these are surrounded by double quotes, which makes it one.  The %~1 on line 40 grabs that argument and outputs the argument without quotes.