tag:blogger.com,1999:blog-91294631972330573632024-03-13T10:47:16.397-05:00Stuart's Semi-Professional BlogI'm an engineer who doesn't care for a lot of fluff for fluff's sake.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.comBlogger207125tag:blogger.com,1999:blog-9129463197233057363.post-79819156492245527412019-12-10T12:50:00.001-06:002019-12-10T12:50:25.684-06:00Sending sysLog via PythonI had someone ask me the other day if we could ingest alarms into LogicMonitor from their app. For example, if the app had an exception raised, they'd like to send a message to LogicMonitor to open an alarm. The easiest way I could think of doing this was to have a standalone Python script that sends data via Syslog to LM. This is one way this could be done. I haven't tested this, but it's simple enough to see that it should work:<br />
<br />
import logging<br />
import logging.handlers<br />
import sys<br />
my_logger = logging.getLogger('MyLogger')<br />
my_logger.setLevel(logging.INFO)<br />
destIpAddress = sys.argv[1]<br />
destPort = sys.argv[2]<br />
handler = logging.handlers.SysLogHandler(address = (ipAddress,514))<br />
my_logger.addHandler(handler)<br />
my_logger.info(sys.argv[3:].join(" "))<br />
<br />
You'd call it like this:<br />
> send_syslog.py 192.168.25.64 514 My application had an error in the widget creator service.<br />
<div>
<br /></div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-33665501927503308922019-11-01T12:36:00.000-05:002020-03-27T16:35:54.657-05:00Counting Resources Per GroupThis post is now deprecated. The updated content is found <a href="https://github.com/sweenig/lmcommunity/tree/master/Billing">here</a>.<br />
<br />
<a name='more'></a><br /><br />
This week I was able to revisit something I'd made for a previous customer, and this time I think I made it even better.<br />
<br />
In both cases, the customers were MSPs who wanted to generate some statistics around how many objects were in particular groups. They wanted to easily bill their customers for services delivered. The first iteration worked just fine, but ended up being difficult to implement for the other customer because the groups were not only named different, but the group structure itself was nothing similar.<br />
<br />
<h2>
<a href="https://www.blogger.com/null" id="weenig_warning" name="weenig_warning">The Datasource Itself</a></h2>
You'll need the datsource file imported into your portal for any of this to work for you. I'm not publishing the datasource configuration file, nor the dashboard configuration file yet, for several reasons. Once I do, I'll update it here.<br />
<br />
The second iteration was an opportunity to innovate so I stuck to the "S" principle in <a href="https://scotch.io/bar-talk/s-o-l-i-d-the-first-five-principles-of-object-oriented-design">SOLID</a>: Single responsibility. I decided that my datasource shouldn't try to do very specific things regarding groups. The end result is that I started by creating a datasource that pulls a list of all groups and gets the counts for every group. This would put all the data in the hands of the user and let them filter out the things that aren't needed.<br />
<br />
<h4>
Filtering to only report on certain groups</h4>
In order to make filtering easier, I added two instance level properties to the datasource definition: depth and fullPath. Before I go into that, let's see what an example folder structure might look like. This example will be referenced throughout this post.<br />
<br />
<h4>
Example Folder Structure</h4>
<pre>/ (root)
├── Clients
│ ├── Acme
│ │ ├── Collectors
│ │ ├── Network
│ │ ├── Security
│ │ ├── Server
│ │ └── Virtual Machines
│ └── NetQoS
│ ├── Collectors
│ ├── Network
│ ├── Server
│ └── Virtual Machines
├── Devices By Type
├── Minimal Monitoring
└── My Devices
├── Office A
└── Office B</pre>
<br />
<h4>
fullPath</h4>
fullPath contains the actual path of the folder. This can be used in a filter to only include folders under certain other folders. For example, since I know my customers' devices are sorted into the Acme and NetQoS folders, I could add a filter like this:<br />
<pre></pre>
<pre>auto.fullPath CONTAIN "Clients/"</pre>
<br />
This would limit discovery to only those folders containing the string "Clients/", which are the following:<br />
<br />
<pre>Clients/Acme
Clients/Acme/Collectors
Clients/Acme/Network
Clients/Acme/Security
Clients/Acme/Server
Clients/Acme/Virtual Machines
Clients/NetQoS/
Clients/NetQoS/Collectors
Clients/NetQoS/Network
Clients/NetQoS/Server
Clients/NetQoS/Virtual Machines</pre>
<br />
This gets us halfway to where we want to be. It included the main client folders (Acme and NetQoS), but also included the groups under them (children of Acme/NetQoS, grandchildren of Clients, and descendants of the root).<br />
<br />
<h4>
Depth</h4>
Depth refers to the distance from the root of each folder. Setting a filter like this:<br />
<br />
auto.depth == 2<br />
<br />
would get the following folders:<br />
<br />
Clients/Acme<br />
Clients/NetQoS<br />
My Devices/Office A<br />
My Devices/Office B<br />
<br />
Combining the two folders would get only the main client folders (Acme & NetQoS). At any rate, any combination of these two filters should allow the user to filter down to just the folders for which we want counts. If no filters were applied, it would give data on each and every group.<br />
<br />
<h3>
The Data</h3>
The data comes from the LM API. Normally, data in LM is collected from a target and stored under the resource in LM that corresponds to that target. This can be most directly/simply done by adding your LM portal to LM as a device. In my case, since my portal is lmstuartweenig, I added "lmstuartweenig.logicmonitor.com" as a device, using the expert mode. I then disabled everything that was discovered by default (because I don't really need to monitor that, to each his own).<br />
<br />
Alternatively, you could avoid adding another resource into your portal by having the datasource collect the data from the API, but storing/associating the data with one of the collectors. This saves you 1 license. Either way, you need to have a resource in your resource tree to which which the data can be associated. In my case, I added my portal as a resource. The only difference is whether or not you burn a license.<br />
<br />
Once you've identified which resource with which the data will be associated, you need to add a few properties. These properties serve two purposes: 1) it signals LM to enable the datasource and where to associate the data and 2) it provides some key information that the datasource will need to access the API. <a href="https://www.logicmonitor.com/support/devices/adding-managing-devices/device-properties/">Add the following properties</a> to the device:<br />
<center>
<table style="border-collapse: collapse; border: 1px solid black;">
<tbody>
<tr><td style="border: 1px solid black;">billing_api.account</td><td style="border: 1px solid black;">This is the account name for your portal. In my case, since my portal address is lmstuartweenig.logicmonitor.com, I'll store "lmstuartweenig" in this property.</td></tr>
<tr><td style="border: 1px solid black;">billing_api.id</td><td style="border: 1px solid black;"><a href="https://www.logicmonitor.com/support/settings/users-and-roles/api-tokens/">Generate an API token</a> and store the ID here.</td></tr>
<tr><td style="border: 1px solid black;">billing_api.pass</td><td style="border: 1px solid black;">Store the API Key here.</td></tr>
</tbody></table>
</center>
<br />
Once this is done, you can wait 15 minutes and the datasource should have discovered the groups and started collecting data. At this point, you may want to go to the datasource definition and tweak the filters for your purposes.<br />
<br />
<h4>
Datapoints</h4>
The following datapoints are gathered for each folder:<br />
<center>
<table style="border-collapse: collapse; border: 1px solid black;"><tbody>
<tr><td style="border: 1px solid black;">Datapoint</td><td style="border: 1px solid black;">Description</td></tr>
<tr><td style="border: 1px solid black;">host_count</td><td style="border: 1px solid black;">The total count of all hosts in the group (including hosts in sub-groups)</td></tr>
<tr><td style="border: 1px solid black;">gcp_count</td><td style="border: 1px solid black;">How many GCP devices (not sure if it includes sub-groups, I think it does)</td></tr>
<tr><td style="border: 1px solid black;">azure_count</td><td style="border: 1px solid black;">How many Azure devices (not sure if it includes sub-groups, I think it does)</td></tr>
<tr><td style="border: 1px solid black;">aws_count</td><td style="border: 1px solid black;">How many AWS devices (not sure if it includes sub-groups, I think it does)</td></tr>
<tr><td style="border: 1px solid black;">Total_Count</td><td style="border: 1px solid black;">Simply the sum of host_count, gcp_count, azure_count, aws_count</td></tr>
</tbody></table>
</center>
<br />
<h4>
Adding Billing Information</h4>
Since one of the original purposes of this data source was to bill customers, it made sense to build in some billing capability. This component is optional and will show "No Data" until it is configured (which I'll detail below).<br />
Since the resulting set of folders normally would represent customers, and since some customers sometimes have different prices for services (gold customers may pay more, for example), and since different type of things could have different costs, you can add a few properties to have the data source calculate some costs as part of the data source with high flexibility.<br />
These properties can be added at the device level. Doing so, would apply the same unit cost to every customer unless a unit cost was applied at the instance level. <b>Let me be clear:</b> the device and instances I'm talking about here are the ones to which the data is associated. Putting these properties on the groups you want to track will have no effect. In my case I added these properties to the lmstuartweenig.logicmonitor.com resource, then added some overrides on the instances under the "Resource Group Member Counts" and "Website Group Member Counts" datasources.<br />
<br />
Specifically, you can add the following properties:<br />
<center>
<table style="border-collapse: collapse; border: 1px solid black;"><tbody>
<tr><td style="border: 1px solid black;">Property</td><td style="border: 1px solid black;">Description</td></tr>
<tr><td style="border: 1px solid black;">billing_api.aws_cost</td><td style="border: 1px solid black;">Unit cost of AWS instances</td></tr>
<tr><td style="border: 1px solid black;">billing_api.azure_cost</td><td style="border: 1px solid black;">Unit cost of Azure instances</td></tr>
<tr><td style="border: 1px solid black;">billing_api.gcp_cost</td><td style="border: 1px solid black;">Unit cost of GCP instances</td></tr>
<tr><td style="border: 1px solid black;">billing_api.resource_cost</td><td style="border: 1px solid black;">Unit cost of devices/resources</td></tr>
<tr><td style="border: 1px solid black;">billing_api.website_cost</td><td style="border: 1px solid black;">Unit cost of a website check</td></tr>
</tbody></table>
</center>
<br />
Once any number of these properties are added, the value of these properties will feed into new datapoints, populated using the formulas shown below:<br />
<center>
<table style="border-collapse: collapse; border: 1px solid black;"><tbody>
<tr><td style="border: 1px solid black;">Datapoint</td><td style="border: 1px solid black;">Formula</td></tr>
<tr><td style="border: 1px solid black;">aws_total_cost</td><td style="border: 1px solid black;">aws_count X billing_api.aws_cost</td></tr>
<tr><td style="border: 1px solid black;">azure_total_cost</td><td style="border: 1px solid black;">azure_count X billing_api.azure_cost</td></tr>
<tr><td style="border: 1px solid black;">gcp_total_cost</td><td style="border: 1px solid black;">gcp_count X billing_api.gcp_cost</td></tr>
<tr><td style="border: 1px solid black;">host_total_cost</td><td style="border: 1px solid black;">host_count X billing_api.resource_cost</td></tr>
<tr><td style="border: 1px solid black;">website_cost</td><td style="border: 1px solid black;">numOfWebsites X billing_api.website_cost</td></tr>
</tbody></table>
</center>
<br />
Using these newly populated datapoints, you can create a dashboard to display all these metrics along with their corresponding cost. Additionally or alternatively, you could create reports containing all the billing data. I recommend dashboards sent out as reports because they are prettier and easier to digest than a flat table of data. Dashboards also allow you to simplify the verbiage.<br />
<br />
One other little caveat: this datasource by default only pulls 50 groups. If you have more than 50 groups in your resource and/or website structure, you'll need to add a property called billing_api.query_limit with a value greater than the total number of folders in the larger structure, recursively.<br />
<br />
<h3>
Creating the Dashboard</h3>
There's a lot that goes into creating a dashboard. Luckily, I've done all the work for you so you can just <a href="https://www.blogger.com/blogger.g?blogID=9129463197233057363#weenig_warning">import</a> the dashboard. However, when importing, you need to understand how the data is filtered down to one customer. The dashboard makes use of <a href="https://www.logicmonitor.com/support/dashboards-and-widgets/managing-dashboards/how-are-dashboards-created/">tokens</a>, which are just variables that you create on the dashboard level. Particularly, you need to create the ##customer## token where the value is the name of the customer. It's important that the value of this token match the name of the folder containing this customer's resources <b>and</b> websites. Meaning that in order for website data to show up along with resource data, the folders should be named the same (technically they don't have to be the same because the filter looks for any discovered folders <i>containing</i> the value of ##customer##).<br />
<br />
In my case, my website folder structure is a bit simpler than my resource group structure:<br />
<pre>Websites/
├── Acme
├── My\ Stuff
└── NetQoS</pre>
<br />
However, since I have a folder for Acme and a folder for NetQoS and the ##customer## token can cover both my resources and my websites. After selecting the dashboard definition file to import, All I do is change the dashboard name and update the token. As long as the data is there, it should show everything. Every widget (and you can see the configuration of each one) makes reference to the token so they all filter to only show data concerning the customer of interest.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-q5gTl2nn6xA/Xbxlt_FVBmI/AAAAAAABsCo/rsf46UyPtxMaxTTTCkqMJjQQaPCoBU4vACLcBGAsYHQ/s1600/Screen%2BShot%2B2019-11-01%2Bat%2B12.04.17%2BPM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="896" data-original-width="696" height="320" src="https://1.bp.blogspot.com/-q5gTl2nn6xA/Xbxlt_FVBmI/AAAAAAABsCo/rsf46UyPtxMaxTTTCkqMJjQQaPCoBU4vACLcBGAsYHQ/s320/Screen%2BShot%2B2019-11-01%2Bat%2B12.04.17%2BPM.png" width="248" /></a></div>
<div style="text-align: left;">
<br /></div>
<h3 style="text-align: left;">
Additional (Accidental) Uses</h3>
<div style="text-align: left;">
One interesting discovery is that if you're only presenting the data through the dashboard, you can do all the filtering on the dashboard level. Meaning: you could actually discover and track every group's metrics, but by properly using the dashboard token, you can filter out only to the one group you're interested in showing.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Another discovery: If you choose your token wisely, you can get multiple customer's data together on one dashboard to get a breakdown of the counts per customer. The token would have to be one that would match on all the customer groups that I want to show. Luckily in my case, I have my customer groups under "Clients". The token would then be "Clients-" (the "/" in the path is replaced with a "-" when the instance is named). </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-fVccg9qVbq4/XbxofOdynqI/AAAAAAABsC0/H6mplFsI8k8FKm2dUScIIxByn0JebTsAgCLcBGAsYHQ/s1600/Screen%2BShot%2B2019-11-01%2Bat%2B12.16.28%2BPM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="879" data-original-width="1544" height="364" src="https://1.bp.blogspot.com/-fVccg9qVbq4/XbxofOdynqI/AAAAAAABsC0/H6mplFsI8k8FKm2dUScIIxByn0JebTsAgCLcBGAsYHQ/s640/Screen%2BShot%2B2019-11-01%2Bat%2B12.16.28%2BPM.png" width="640" /></a></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Notice in the legends of the pie charts, that it shows one slice for NetQoS' resources and a different slice for Acme's resources. FYI: ignore the oddness in the top trend plots as a configuration change right before this snapshot caused some historical data to be lost and start gathering again). </div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-65474090368566063202019-09-03T15:12:00.001-05:002019-10-08T10:23:26.209-05:00Groovy SNMPwalk Helper Functionstl;dr - <a href="https://github.com/sweenig/monitoring-recipes/tree/master/DataSources/Groovy/SNMP/Examples">Go here.</a><br />
<br />
I've been doing a lot of SNMP polling through Groovy lately. One of the methods I use most often is Snmp.walkAsMap(), which returns a map that looks like this:<br />
<pre>[8.6:2, 4.10:1500, 8.7:1, 8.8:1, 4.12:1500, 4.14:1500, 4.16:1500, 3.44:6, 22.5:0.0, 22.4:0.0,
22.3:0.0, 22.2:0.0, 22.1:0.0, 22.8:0.0, 22.7:0.0, 22.6:0.0, 17.16:164136, 16.44:6639007, 17.12:8634086,
18.8:0, 17.14:8745523, 18.7:0, 17.10:21076, 10.6:0, 10.5:0, 10.4:0, 10.3:0, 10.2:3968863511, 10.1:8980402,
22.44:0.0, 18.6:0, 18.5:0, 7.1:1, 18.4:0, 7.2:1, 18.3:0, 7.3:1, 18.2:0, 7.4:1, 18.1:0, 7.5:1, 10.8:1684285,
7.6:2, 10.7:4181064938, 7.7:1, 7.8:1, 15.44:0, 16.12:2114735865, 16.10:4077101, 16.16:16467729,
16.14:2361323502, 22.14:0.0, 22.16:0.0, 9.44:2 days, 14:54:42.04, 21.6:0, 21.5:0, 21.4:0, 21.3:0,
21.2:0, 21.1:0, 22.10:0.0, 22.12:0.0, 21.44:0, 21.8:0, 21.7:0, 5.10:4294967295, 4.44:1500, 5.12:4294967295,
5.14:4294967295, 11.10:0, 5.16:4294967295, 10.44:1620014, 11.12:12487957, 17.8:30812, 11.14:12673362,
17.7:17502386, 6.1:, 17.6:0, 6.2:90:e6:ba:59:1b:18, 11.16:136053, 17.5:0, 6.3:00:50:56:c0:00:01,
17.4:20226, 6.4:00:50:56:c0:00:08, 17.3:20229, 6.5:52:54:00:9b:4b:e0, 17.2:37200831, 6.6:52:54:00:9b:4b:e0,
17.1:44466, 6.7:02:42:39:1b:cc:e3, 6.8:02:42:55:44:3f:06, 10.10:0, 20.7:0, 20.6:0, 20.5:0, 20.4:0, 20.3:0,
20.2:0, 20.1:0, 10.12:30849041, 10.14:131456967, 10.16:77954842, 20.8:0, 5.1:10000000, 16.8:17045376,
5.2:100000000, 16.7:189459674, 5.3:0, 16.6:0, 5.4:0, 16.5:0, 5.5:0, 16.4:0, 5.6:10000000, 16.3:0, 5.7:0,
16.2:1009509757, 5.8:0, 16.1:8980402, 6.12:4e:c1:4e:e6:07:90, 5.44:4294967295, 6.14:0a:2c:4f:3a:eb:5c,
6.16:12:db:e0:82:ca:a2, 19.44:0, 6.10:06:31:50:31:a5:fb, 15.12:0, 14.44:0, 15.10:0, 21.16:0, 15.16:0,
21.14:0, 15.14:0, 20.44:0, 21.12:0, 1.12:12, 1.10:10, 15.1:0, 4.1:65536, 4.2:1500, 4.3:1500, 15.8:0,
21.10:0, 4.4:1500, 15.7:0, 4.5:1500, 15.6:0, 4.6:1500, 15.5:0, 4.7:1500, 15.4:0, 4.8:1500, 15.3:0,
15.2:0, 14.10:0, 20.16:0, 14.14:0, 20.14:0, 13.44:0, 14.12:0, 20.12:0, 1.16:16, 1.14:14, 20.10:0, 14.16:0,
6.44:62:d0:3f:14:ef:85, 7.12:1, 7.14:1, 7.16:1, 7.10:1, 14.2:0, 14.1:0, 3.1:24, 3.2:6, 3.3:6, 3.4:6, 3.5:6,
14.8:0, 3.6:6, 14.7:0, 3.7:6, 14.6:0, 3.8:6, 14.5:0, 14.4:0, 14.3:0, 2.14:veth9a64fa1, 2.12:veth2bc8fbd,
1.44:44, 2.10:veth92ca0b0, 19.14:0, 19.16:0, 19.10:0, 19.12:0, 18.44:0, 13.3:0, 13.2:0, 13.1:0, 2.1:lo,
2.2:NVIDIA Corporation MCP77 Ethernet, 2.16:veth497df7f, 2.3:vmnet1, 2.4:vmnet8, 2.5:virbr0, 2.6:virbr0-nic,
2.7:br-6a2604a91ac1, 13.8:0, 2.8:docker0, 13.7:0, 13.6:0, 13.5:0, 13.4:0, 18.16:0, 8.16:1, 8.14:1,
17.44:18857, 18.12:0, 18.14:0, 18.10:0, 8.12:1, 7.44:1, 8.10:1, 13.10:0, 12.44:0, 13.14:0, 13.12:0,
3.14:6, 2.44:vethd608288, 3.12:6, 3.10:6, 12.4:0, 12.3:0, 12.2:2662, 12.1:0, 1.1:1, 1.2:2, 1.3:3, 1.4:4,
1.5:5, 1.6:6, 1.7:7, 1.8:8, 13.16:0, 9.1:0:00:00.00, 12.8:0, 9.2:0:00:00.00, 12.7:0, 9.3:0:00:06.31, 12.6:0,
9.4:0:00:06.31, 12.5:0, 9.5:0:00:09.32, 9.6:0:00:12.32, 9.7:3:03:43.67, 9.8:0:00:18.32, 12.10:0, 11.44:4506,
12.12:0, 3.16:6, 12.14:0, 12.16:0, 9.16:3:03:43.67, 9.14:3:03:43.67, 19.8:0, 19.7:0, 19.6:0, 8.44:1,
9.12:3:03:43.67, 9.10:0:00:18.32, 11.5:0, 11.4:0, 11.3:0, 11.2:28692938, 11.1:44466, 19.5:0, 19.4:0,
19.3:0, 8.1:1, 19.2:0, 8.2:1, 19.1:0, 8.3:1, 11.8:6882, 8.4:1, 11.7:25297372, 8.5:2, 11.6:0]</pre>
<br />
<br />
<br />
This great and usable, but I found myself constantly wanting to display the data easier. I also wanted the data to be structured a little more hierarchically, grouping by row in the SNMP table. I also wanted to make it easier to address individual pieces of the data. I built a couple helper functions that transform the data and make it easier to address. They can be found <a href="https://github.com/sweenig/monitoring-recipes/tree/master/DataSources/Groovy/SNMP/Examples">here</a>.<br />
<br />
<br />
<br />
<br />
The output of the snmpMapToTable() function looks like this:<br />
<pre>[6:[8:2, 22:0.0, 10:0, 18:0, 7:2, 21:0, 17:0, 6:52:54:00:9b:4b:e0, 20:0, 16:0, 5:10000000, 15:0, 4:1500,
3:6, 14:0, 2:virbr0-nic, 13:0, 1:6, 12:0, 9:0:00:12.32, 19:0, 11:0], 10:[4:1500, 17:21076, 16:4077101, 22:0.0,
5:4294967295, 11:0, 10:0, 6:06:31:50:31:a5:fb, 15:0, 1:10, 21:0, 14:0, 20:0, 7:1, 2:veth92ca0b0, 19:0, 18:0,
8:1, 13:0, 3:6, 12:0, 9:0:00:18.32], 7:[8:1, 22:0.0, 18:0, 10:4181064938, 7:1, 21:0, 17:17502386,
6:02:42:39:1b:cc:e3, 20:0, 16:189459674, 5:0, 15:0, 4:1500, 14:0, 3:6, 2:br-6a2604a91ac1, 13:0, 1:7,
12:0, 9:3:03:43.67, 19:0, 11:25297372], 8:[8:1, 22:0.0, 18:0, 10:1684285, 7:1, 21:0, 17:30812,
6:02:42:55:44:3f:06, 20:0, 16:17045376, 5:0, 15:0, 4:1500, 14:0, 3:6, 13:0, 2:docker0, 1:8, 12:0,
9:0:00:18.32, 19:0, 11:6882], 12:[4:1500, 17:8634086, 16:2114735865, 22:0.0, 5:4294967295,
11:12487957, 10:30849041, 6:4e:c1:4e:e6:07:90, 15:0, 21:0, 1:12, 14:0, 20:0, 7:1, 2:veth2bc8fbd,
19:0, 18:0, 8:1, 13:0, 3:6, 12:0, 9:3:03:43.67], 14:[4:1500, 17:8745523, 16:2361323502, 22:0.0,
5:4294967295, 11:12673362, 10:131456967, 6:0a:2c:4f:3a:eb:5c, 21:0, 15:0, 14:0, 20:0, 1:14, 7:1,
2:veth9a64fa1, 19:0, 8:1, 18:0, 13:0, 3:6, 12:0, 9:3:03:43.67], 16:[4:1500, 17:164136, 16:16467729,
22:0.0, 5:4294967295, 11:136053, 10:77954842, 6:12:db:e0:82:ca:a2, 21:0, 15:0, 20:0, 1:16, 14:0,
7:1, 19:0, 2:veth497df7f, 18:0, 8:1, 13:0, 3:6, 12:0, 9:3:03:43.67], 44:[3:6, 16:6639007, 22:0.0,
15:0, 9:2 days, 14:54:42.04, 21:0, 4:1500, 10:1620014, 5:4294967295, 19:0, 14:0, 20:0,
13:0, 6:62:d0:3f:14:ef:85, 1:44, 18:0, 17:18857, 7:1, 12:0, 2:vethd608288, 11:4506, 8:1], 5:[22:0.0,
10:0, 18:0, 7:1, 21:0, 17:0, 6:52:54:00:9b:4b:e0, 20:0, 16:0, 5:0, 4:1500, 15:0, 3:6, 14:0, 2:virbr0,
13:0, 1:5, 12:0, 9:0:00:09.32, 11:0, 19:0, 8:2], 4:[22:0.0, 10:0, 18:0, 7:1, 21:0, 17:20226,
6:00:50:56:c0:00:08, 20:0, 5:0, 16:0, 4:1500, 15:0, 3:6, 14:0, 2:vmnet8, 13:0, 12:0, 1:4, 9:0:00:06.31,
11:0, 19:0, 8:1], 3:[22:0.0, 10:0, 18:0, 7:1, 21:0, 6:00:50:56:c0:00:01, 17:20229, 20:0, 5:0, 16:0,
4:1500, 15:0, 3:6, 14:0, 13:0, 2:vmnet1, 12:0, 1:3, 9:0:00:06.31, 11:0, 19:0, 8:1], 2:[22:0.0,
10:3968863511, 7:1, 18:0, 21:0, 6:90:e6:ba:59:1b:18, 17:37200831, 20:0, 5:100000000, 16:1009509757,
4:1500, 15:0, 14:0, 3:6, 13:0, 2:NVIDIA Corporation MCP77 Ethernet, 12:2662, 1:2, 9:0:00:00.00,
11:28692938, 19:0, 8:1], 1:[22:0.0, 10:8980402, 7:1, 18:0, 21:0, 6:, 17:44466, 20:0, 5:10000000,
16:8980402, 15:0, 4:65536, 14:0, 3:24, 13:0, 2:lo, 12:0, 1:1, 9:0:00:00.00, 11:44466, 8:1, 19:0]]</pre>
<br />
<br />
<br />
<br />
<br />
It may not look much better, but if you add some carriage returns and tabs you get this:<br />
<pre>[
6:[
8:2, 22:0.0, 10:0, 18:0, 7:2, 21:0, 17:0, 6:52:54:00:9b:4b:e0,
20:0, 16:0, 5:10000000, 15:0, 4:1500, 3:6, 14:0, 2:virbr0-nic,
13:0, 1:6, 12:0, 9:0:00:12.32, 19:0, 11:0],
10:[
4:1500, 17:21076, 16:4077101, 22:0.0, 5:4294967295, 11:0, 10:0,
6:06:31:50:31:a5:fb, 15:0, 1:10, 21:0, 14:0, 20:0, 7:1, 2:veth92ca0b0,
19:0, 18:0, 8:1, 13:0, 3:6, 12:0, 9:0:00:18.32],
7:[
8:1, 22:0.0, 18:0, 10:4181064938, 7:1, 21:0, 17:17502386, 6:02:42:39:1b:cc:e3,
20:0, 16:189459674, 5:0, 15:0, 4:1500, 14:0, 3:6, 2:br-6a2604a91ac1, 13:0, 1:7,
12:0, 9:3:03:43.67, 19:0, 11:25297372],
8:[
8:1, 22:0.0, 18:0, 10:1684285, 7:1, 21:0, 17:30812, 6:02:42:55:44:3f:06, 20:0,
16:17045376, 5:0, 15:0, 4:1500, 14:0, 3:6, 13:0, 2:docker0, 1:8, 12:0, 9:0:00:18.32,
19:0, 11:6882],
12:[
4:1500, 17:8634086, 16:2114735865, 22:0.0, 5:4294967295, 11:12487957, 10:30849041,
6:4e:c1:4e:e6:07:90, 15:0, 21:0, 1:12, 14:0, 20:0, 7:1, 2:veth2bc8fbd, 19:0, 18:0, 8:1,
13:0, 3:6, 12:0, 9:3:03:43.67],
14:[
4:1500, 17:8745523, 16:2361323502, 22:0.0, 5:4294967295, 11:12673362, 10:131456967,
6:0a:2c:4f:3a:eb:5c, 21:0, 15:0, 14:0, 20:0, 1:14, 7:1, 2:veth9a64fa1, 19:0, 8:1, 18:0,
13:0, 3:6, 12:0, 9:3:03:43.67],
16:[
4:1500, 17:164136, 16:16467729, 22:0.0, 5:4294967295, 11:136053, 10:77954842,
6:12:db:e0:82:ca:a2, 21:0, 15:0, 20:0, 1:16, 14:0, 7:1, 19:0, 2:veth497df7f, 18:0,
8:1, 13:0, 3:6, 12:0, 9:3:03:43.67],
44:[
3:6, 16:6639007, 22:0.0, 15:0, 9:2 days, 14:54:42.04, 21:0, 4:1500, 10:1620014,
5:4294967295, 19:0, 14:0, 20:0, 13:0, 6:62:d0:3f:14:ef:85, 1:44, 18:0, 17:18857,
7:1, 12:0, 2:vethd608288, 11:4506, 8:1],
5:[
22:0.0, 10:0, 18:0, 7:1, 21:0, 17:0, 6:52:54:00:9b:4b:e0, 20:0, 16:0, 5:0, 4:1500,
15:0, 3:6, 14:0, 2:virbr0, 13:0, 1:5, 12:0, 9:0:00:09.32, 11:0, 19:0, 8:2],
4:[
22:0.0, 10:0, 18:0, 7:1, 21:0, 17:20226, 6:00:50:56:c0:00:08, 20:0, 5:0, 16:0,
4:1500, 15:0, 3:6, 14:0, 2:vmnet8, 13:0, 12:0, 1:4, 9:0:00:06.31, 11:0, 19:0, 8:1],
3:[
22:0.0, 10:0, 18:0, 7:1, 21:0, 6:00:50:56:c0:00:01, 17:20229, 20:0, 5:0, 16:0, 4:1500,
15:0, 3:6, 14:0, 13:0, 2:vmnet1, 12:0, 1:3, 9:0:00:06.31, 11:0, 19:0, 8:1],
2:[
22:0.0, 10:3968863511, 7:1, 18:0, 21:0, 6:90:e6:ba:59:1b:18, 17:37200831, 20:0,
5:100000000, 16:1009509757, 4:1500, 15:0, 14:0, 3:6, 13:0,
2:NVIDIA Corporation MCP77 Ethernet, 12:2662, 1:2, 9:0:00:00.00, 11:28692938, 19:0, 8:1],
1:[
22:0.0, 10:8980402, 7:1, 18:0, 21:0, 6:, 17:44466, 20:0, 5:10000000, 16:8980402, 15:0,
4:65536, 14:0, 3:24, 13:0, 2:lo, 12:0, 1:1, 9:0:00:00.00, 11:44466, 8:1, 19:0]
]</pre>
<br />
<br />
<br />
<br />
As you can see, it's getting easier to see what's going on. At this point, it'd be nice to order by instance ID and also order by metric ID. One of the helper functions I built does just that, pprintSnmpWalkTable:<br />
<pre>Wildvalue: 1:
Data sorted by column ID
1.##WILDVALUE##: 1
2.##WILDVALUE##: lo
3.##WILDVALUE##: 24
4.##WILDVALUE##: 65536
5.##WILDVALUE##: 10000000
6.##WILDVALUE##:
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 0:00:00.00
10.##WILDVALUE##: 8980402
11.##WILDVALUE##: 44466
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 8980402
17.##WILDVALUE##: 44466
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 2:
Data sorted by column ID
1.##WILDVALUE##: 2
2.##WILDVALUE##: NVIDIA Corporation MCP77 Ethernet
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 100000000
6.##WILDVALUE##: 90:e6:ba:59:1b:18
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 0:00:00.00
10.##WILDVALUE##: 3968863511
11.##WILDVALUE##: 28692938
12.##WILDVALUE##: 2662
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 1009509757
17.##WILDVALUE##: 37200831
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 3:
Data sorted by column ID
1.##WILDVALUE##: 3
2.##WILDVALUE##: vmnet1
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 0
6.##WILDVALUE##: 00:50:56:c0:00:01
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 0:00:06.31
10.##WILDVALUE##: 0
11.##WILDVALUE##: 0
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 0
17.##WILDVALUE##: 20229
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 4:
Data sorted by column ID
1.##WILDVALUE##: 4
2.##WILDVALUE##: vmnet8
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 0
6.##WILDVALUE##: 00:50:56:c0:00:08
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 0:00:06.31
10.##WILDVALUE##: 0
11.##WILDVALUE##: 0
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 0
17.##WILDVALUE##: 20226
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 5:
Data sorted by column ID
1.##WILDVALUE##: 5
2.##WILDVALUE##: virbr0
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 0
6.##WILDVALUE##: 52:54:00:9b:4b:e0
7.##WILDVALUE##: 1
8.##WILDVALUE##: 2
9.##WILDVALUE##: 0:00:09.32
10.##WILDVALUE##: 0
11.##WILDVALUE##: 0
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 0
17.##WILDVALUE##: 0
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 6:
Data sorted by column ID
1.##WILDVALUE##: 6
2.##WILDVALUE##: virbr0-nic
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 10000000
6.##WILDVALUE##: 52:54:00:9b:4b:e0
7.##WILDVALUE##: 2
8.##WILDVALUE##: 2
9.##WILDVALUE##: 0:00:12.32
10.##WILDVALUE##: 0
11.##WILDVALUE##: 0
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 0
17.##WILDVALUE##: 0
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 7:
Data sorted by column ID
1.##WILDVALUE##: 7
2.##WILDVALUE##: br-6a2604a91ac1
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 0
6.##WILDVALUE##: 02:42:39:1b:cc:e3
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 3:03:43.67
10.##WILDVALUE##: 4181064938
11.##WILDVALUE##: 25297372
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 189459674
17.##WILDVALUE##: 17502386
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 8:
Data sorted by column ID
1.##WILDVALUE##: 8
2.##WILDVALUE##: docker0
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 0
6.##WILDVALUE##: 02:42:55:44:3f:06
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 0:00:18.32
10.##WILDVALUE##: 1684285
11.##WILDVALUE##: 6882
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 17045376
17.##WILDVALUE##: 30812
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 10:
Data sorted by column ID
1.##WILDVALUE##: 10
2.##WILDVALUE##: veth92ca0b0
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 4294967295
6.##WILDVALUE##: 06:31:50:31:a5:fb
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 0:00:18.32
10.##WILDVALUE##: 0
11.##WILDVALUE##: 0
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 4077101
17.##WILDVALUE##: 21076
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 12:
Data sorted by column ID
1.##WILDVALUE##: 12
2.##WILDVALUE##: veth2bc8fbd
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 4294967295
6.##WILDVALUE##: 4e:c1:4e:e6:07:90
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 3:03:43.67
10.##WILDVALUE##: 30849041
11.##WILDVALUE##: 12487957
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 2114735865
17.##WILDVALUE##: 8634086
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 14:
Data sorted by column ID
1.##WILDVALUE##: 14
2.##WILDVALUE##: veth9a64fa1
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 4294967295
6.##WILDVALUE##: 0a:2c:4f:3a:eb:5c
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 3:03:43.67
10.##WILDVALUE##: 131456967
11.##WILDVALUE##: 12673362
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 2361323502
17.##WILDVALUE##: 8745523
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 16:
Data sorted by column ID
1.##WILDVALUE##: 16
2.##WILDVALUE##: veth497df7f
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 4294967295
6.##WILDVALUE##: 12:db:e0:82:ca:a2
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 3:03:43.67
10.##WILDVALUE##: 77954842
11.##WILDVALUE##: 136053
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 16467729
17.##WILDVALUE##: 164136
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0
Wildvalue: 44:
Data sorted by column ID
1.##WILDVALUE##: 44
2.##WILDVALUE##: vethd608288
3.##WILDVALUE##: 6
4.##WILDVALUE##: 1500
5.##WILDVALUE##: 4294967295
6.##WILDVALUE##: 62:d0:3f:14:ef:85
7.##WILDVALUE##: 1
8.##WILDVALUE##: 1
9.##WILDVALUE##: 2 days, 14:54:42.04
10.##WILDVALUE##: 1620014
11.##WILDVALUE##: 4506
12.##WILDVALUE##: 0
13.##WILDVALUE##: 0
14.##WILDVALUE##: 0
15.##WILDVALUE##: 0
16.##WILDVALUE##: 6639007
17.##WILDVALUE##: 18857
18.##WILDVALUE##: 0
19.##WILDVALUE##: 0
20.##WILDVALUE##: 0
21.##WILDVALUE##: 0
22.##WILDVALUE##: 0.0</pre>
<br />
<br />
<br />
For me, this is pretty easy to read and find individual values that I'm looking for. The three lines that resulted in the above output were these:<br />
<pre>walkResult = Snmp.walkAsMap(host, Oid, props, timeout)
entryRaw = snmpMapToTable(walkResult)
pprintSnmpWalkTable(entryRaw)
</pre>
<br />
<br />
<br />
Accessing the data is pretty easy now:<br />
<pre>ifEntryRaw.each {wildvalue, data ->
println("""Interface ${wildvalue}:
ifAlias: ${ifXEntryRaw[wildvalue]["1"]}
ifDescr: ${data["2"]}
ifType: ${data["3"]}
""")
}</pre>
<br />
<br />
<br />
This gives us the following output:<br />
<pre>Interface 6:
ifAlias: virbr0-nic
ifDescr: virbr0-nic
ifType: 6
Interface 10:
ifAlias: veth92ca0b0
ifDescr: veth92ca0b0
ifType: 6
Interface 7:
ifAlias: br-6a2604a91ac1
ifDescr: br-6a2604a91ac1
ifType: 6
Interface 8:
ifAlias: docker0
ifDescr: docker0
ifType: 6
Interface 12:
ifAlias: veth2bc8fbd
ifDescr: veth2bc8fbd
ifType: 6
Interface 14:
ifAlias: veth9a64fa1
ifDescr: veth9a64fa1
ifType: 6
Interface 16:
ifAlias: veth497df7f
ifDescr: veth497df7f
ifType: 6
Interface 44:
ifAlias: vethd608288
ifDescr: vethd608288
ifType: 6
Interface 5:
ifAlias: virbr0
ifDescr: virbr0
ifType: 6
Interface 4:
ifAlias: vmnet8
ifDescr: vmnet8
ifType: 6
Interface 3:
ifAlias: vmnet1
ifDescr: vmnet1
ifType: 6
Interface 2:
ifAlias: enp0s10
ifDescr: NVIDIA Corporation MCP77 Ethernet
ifType: 6
Interface 1:
ifAlias: lo
ifDescr: lo
ifType: 24</pre>
<br />Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-78116548377140614912019-07-05T09:05:00.000-05:002019-07-05T09:05:29.823-05:00 Discovering Enumerated Propertiestl;dr - the scripts are <a href="https://github.com/logicmonitor/monitoring-recipes/blob/master/DataSources/Groovy/SNMP/Examples/ActiveDiscovery_with_properties_with_enumeration.groovy">here</a> and <a href="https://github.com/logicmonitor/monitoring-recipes/blob/master/PropertySources/Examples/PropertySource%20with%20enumeration.groovy">here</a>.<br />
<br />
In the same vein as the <a href="https://stuart.weenig.com/2019/07/visualizing-status-codes.html">previous post</a>, this post will talk about SNMP enumeration. While the <a href="https://stuart.weenig.com/2019/07/visualizing-status-codes.html">previous post talked about polling enumerated values and how to display them as intuitively as possible</a>, this post will talk about polling more static values and using them as properties.<br />
<br />
There are two levels of properties that can be obtained via SNMP: device level properties (that pertain to the whole device) and instance level properties (that pertain to a particular thing on the device, of which there may be more than one). Polling those properties is easy; this post will go over how to improve the quality of the data stored so that the data can be intuitively used.<br />
<h4>
Polling properties vs. data points</h4>
<div>
Polling properties is easy. Knowing which properties to poll as properties and which to poll as data points is a separate discussion entirely. Suffice it to say that things that represent a characteristic about an object should be properties and things that represent a behavior should be data points. Depending on the tool, only data points can be alerted upon, so that may influence a decision to make something that would normally be a property also a data point. In other cases, sometimes data points can't be used to influence enablement of monitoring; so sometimes data points need to be properties too.</div>
<div>
Assuming you're past the point where you've looked at the MIB and figured out which OIDs should be properties and which should be data points, the next thing to look at is whether or not the OIDs you will be storing as properties have enumerations.</div>
<div>
<h4>
Enumerations</h4>
</div>
<div>
An enumeration is used in SNMP to keep the <b>simple</b> in <b>Simple</b> Network Management Protocol. Instead of passing back a string comprised of ASCII characters, a map is created that connects a meaningful string to a single integer, which is passed back to the NMS. Passing back a single integer is much simpler than passing back a string of characters of varying length.</div>
<div>
For example, the <a href="http://www.net-snmp.org/docs/mibs/interfaces.html#ifAdminStatus">admin status of an interface</a> is found at <a href="http://oid-info.com/get/1.3.6.1.2.1.2.2.1.7">1.3.6.1.2.1.2.2.1.7</a>. It's not a particularly good example of a property since it lends itself more to a data point, but having it as a property can allow for advanced filtering which may only be available using properties.<br />
<pre class="code">ifAdminStatus OBJECT-TYPE
SYNTAX INTEGER {
up(1), -- ready to pass packets
down(2),
testing(3) -- in some test mode
}
MAX-ACCESS read-write
STATUS current
DESCRIPTION
"The desired state of the interface. The testing(3) state
indicates that no operational packets can be passed. When a
managed system initializes, all interfaces start with
ifAdminStatus in the down(2) state. As a result of either
explicit management action or per configuration information
retained by the managed system, ifAdminStatus is then
changed to either the up(1) or testing(3) states (or remains
in the down(2) state)."
::= { ifEntry 7 }
</pre>
</div>
<div>
You can see in the definition of the OID that it has a custom syntax that allows for one of three values. When this OID is polled, either a 1, a 2, or a 3 is returned. It's up to the NMS to interpret the different values to understand the meaning. If the NMS tool has MIB compilation, this may have a certain level of automation. If it doesn't, interpretation is up to you. Obviously, storing a 1, 2, or 3 as a property could be sufficient, but it would be better to store the interpreted meaning making use by humans easier and more intuitive.<br />
<h4>
Interpretation of an instance property enumeration</h4>
So how do we interpret this? It's actually pretty easy, but it involves some scripting to process the returned data. It also involves some research into the MIB to find out all the meanings. Let's start with a simple case of interfaces. In most cases a MIB will contain a "Table" OID with an "Entry" child OID containing all the instances with whatever OIDs go along with the instances. In the case of interfaces (ignoring the ifXTable), the table is called <a href="http://oid-info.com/get/1.3.6.1.2.1.2.2">"ifTable" at 1.3.6.1.2.1.2.2</a>. You'll see that there is a child OID called <a href="http://oid-info.com/cgi-bin/display?tree=1.3.6.1.2.1.2.2.1">"ifEntry" at 1.3.6.1.2.1.2.2.1</a>. This kind of table typically has an index along with perhaps some properties and data points as columns. Each row is an instance. We're going to ignore the data points for now and focus on the OIDs that would do well stored as properties. MTU(4), speed(5), and MAC address(6) are good items to store as properties. They require no interpretation. However, ifType(3) and ifAdminStatus(7) require some interpretation to be useful.<br />
LogicMonitor's multi-instance datasource can be configured to use a <a href="https://groovy-lang.org/">groovy script (yes, it's a real thing)</a> to do auto-discovery, which is the mechanism that discovers poll instances and sets properties per poll instance. The <a href="https://www.logicmonitor.com/support/datasources/active-discovery/script-active-discovery/">concept</a> is pretty simple, use the groovy SNMP libraries to retrieve the data, use groovy to interpret the data, then just print the data to standard output, one line per poll instance.<br />
I recently wrote <a href="https://github.com/logicmonitor/monitoring-recipes/blob/master/DataSources/Groovy/SNMP/Examples/ActiveDiscovery_with_properties_with_enumeration.groovy">a script to do this</a>. It's all self documented with explanations and sample output and everything. Some points to consider at the following lines:<br />
<ol>
<li value="11">This line defines the address of the Entry table. Everything we will be polling happens to live under this branch of the OID tree.</li>
<li value="14">This is where we define which column of the ifEntry table contains the name that we should be using for each of our instances.</li>
<li value="32">This line shows the information from the MIB added to the script so that the script can interpret the meaning from the returned value for ifAdminStatus</li>
<li>This line shows the information from the MIB added to the script so that the script can interpret the meaning from the returned value for ifOperStatus</li>
<li value="40">This line shows that we're still polling ifMtu as a property, but there is no interpretation available for the value. We could alternatively put ["1500":"Default (1500)"] instead of [:] to tell the script to add some meaning to the most common value of MTU</li>
<li>ifPhysAddress is the MAC address and needs no interpretation</li>
<li value="51">This line shows an interpretation that isn't defined in the MIB. Instead of just passing the raw value through, we can provide our own interpretation of the speed to give some more intuitive values for common speeds. If the speed of the interface isn't in our map, the speed itself will be stored as the value. If it does happen to match on eof</li>
<li value="59">This is where we start to define the enumeration for ifType. Turns out ifType has over 200 different enumerated values. Each of these is defined in the script so that the proper type name can be stored as a property. </li>
<li value="102">Here's where you can see some sample output against a device here in my house. Notice that there's one line for each port (both physical and logical). </li>
<li value="117">This line is a good example showing the interpreted values of ifAdminStatus, ifOperStatus, ifSpeed, and ifType.</li>
</ol>
<h4>
Interpretation of device level properties</h4>
Polling and storing device level properties is a bit simpler mainly because looping through instances is not required. We do have to provide each OID, the name we want the property stored under, and the interpretation, if any. All of this comes from the MIB. The script to do this is <a href="https://github.com/logicmonitor/monitoring-recipes/blob/master/PropertySources/Examples/PropertySource%20with%20enumeration.groovy">here</a>. This example comes from the mGuard MIB, which is from some work I did recently. However, the OIDs can be replaced with any OID from any MIB (notice there's not a baseOID), as long as the OID returns a single value because the script does an SNMP get on that OID.<br />
<h4>
Conclusion</h4>
That's about it. These two scripts can be used to add real meaning to device and instance level properties. The only things that have to change are the data that come from the MIB itself. Perhaps one of these days I'll get around to writing a MIB parser that will output this information for all OIDs in the MIB. Yeah, when I have time.</div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-84367237519813691922019-07-02T11:32:00.000-05:002019-08-23T11:09:34.210-05:00Visualizing Status CodesIf you follow me on <a href="http://www.linkedin.com/in/sweenig" target="_blank">LinkedIn</a>, you will have noticed that I changed jobs and moved from Houston to Austin. I'm now working for <a href="http://www.logicmonitor.com/" target="_blank">LogicMonitor</a> as a <a href="https://sweenig.github.io/resume/#current" target="_blank">Sales/Monitoring Engineer</a>. It's a great job and I'm enjoying it a lot more than previous jobs. One of the advantages of this job is that I should have much more opportunity, desire, and content for blog posts.<br />
<br />
If this is your first time here, know that this blog is not written for you. It's written for me. I increasingly need more and more reminders of how to do things. That goes especially for things that I devise since no one else knows it unless I tell them. This blog is primarily a place for me to keep those things written down.<br />
<br />
Anyway, on to this blog post. LogicMonitor monitors IT infrastructure. After collecting data through various mechanisms, it stores the data in a big database in the cloud and then provides a cloud hosted front end website to display the data. Part of the display is graphs. Many times, the metrics being graphed lend themselves to being plotted on a <a href="https://en.wikipedia.org/wiki/Cartesian_coordinate_system" target="_blank">Cartesian coordinated</a> graph. However, sometimes the metric being polled is a status code. A good example of this is license status on Sophos' XG Firewall. This metric is found at <a href="http://oid-info.com/get/1.3.6.1.4.1.21067.2.1.3.4.1">.1.3.6.1.4.1.21067.2.1.3.4.1</a>.<br />
<pre class="code">asSubStatus OBJECT-TYPE
SYNTAX SubscriptionStatusType
MAX-ACCESS read-only
STATUS current
DESCRIPTION " "
::= { liAntispam 1 }
</pre>
The syntax is "SubscriptionStatusType", which is an enumerated type meaning that only a number is returned, but that number has a meaning depending on the different values returned. Looking at the syntax definition in the MIB will help illustrate:<br />
<pre class="code">SubscriptionStatusType ::= TEXTUAL-CONVENTION
STATUS current
DESCRIPTION "enumerated type for subscription status"
SYNTAX INTEGER {
trial ( 1 ),
unsubscribed ( 2 ),
subscribed ( 3 ),
expired ( 4 )
}
</pre>
So each different value returned indicates a particular state of the license subscription. It's not like a percentage where 100% is good and 0% is bad and there might be values in between. It's not like a rate, where a high number is fast and a low number is slow. It only has discreet values and values in between don't actually have any meaning.<br />
Normally, without putting in much effort, someone might easily create a graph that just plots this number, putting time on the x-axis and the value retuned on the y-axis. This results in what you see here:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Htza1q7nRlI/XRt17Ovzw_I/AAAAAAABmF0/aKXICH4Tq-EEqX67F_fMa8ydSKmf3W8OACLcBGAs/s1600/Screen%2BShot%2B2019-07-02%2Bat%2B10.18.51%2BAM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="780" data-original-width="1288" height="386" src="https://1.bp.blogspot.com/-Htza1q7nRlI/XRt17Ovzw_I/AAAAAAABmF0/aKXICH4Tq-EEqX67F_fMa8ydSKmf3W8OACLcBGAs/s640/Screen%2BShot%2B2019-07-02%2Bat%2B10.18.51%2BAM.png" width="640" /></a></div>
As you can see, it's not very helpful. It's a flat line because the status has been the same for the entire time range. That's ok. However, there's no real indicator of meaning. Some effort was made to add the meaning to the data point description (which appears in the tooltip). However, the enumeration is so long that it doesn't really fit in the tooltip. What does a 3 mean? Also, it's not illustrated here, but what happens if the value changes from 3 to 4? There would be two flat lines, one at 3 before the change and one at 4 after the change. But what would be shown at the change? Would it be a vertical line from 3 to 4? Would it be slightly slanted? Also not illustrated here, but what happens when larger timeframes are chosen and values are aggregated together (most often using an average)? Imagine that line at 3 that transitions to 4. What if that happened in middle of the quarter and you viewed it at the end of the quarter? If this status was polled every hour, that would mean 2190 data points to display! <a href="https://guides.library.duke.edu/datavis/topten" target="_blank">That's too many</a>. Almost every graphing solution would attempt to decrease the data points by grouping points and averaging every group. In the case of a quarterly timeframe, it might simplify by averaging all data points for a single day together. This could be fine for most days, except for the one where there was a change. That would show an average of 3's and 4's, yielding a value of 3.5. WTH does 3.5 mean? It gets worse if you have a 2 that transitions to a 3 which then later transitions to a 4. You could end up with an average of 3, indicating no problem at all!?!? It's not intuitive; and <a href="https://en.wikipedia.org/wiki/The_Design_of_Everyday_Things" target="_blank">graphs need to be intuitive</a>.<br />
<h4>
So, what do we do?</h4>
Well, we might be tempted to normalize the data. This is actually a very good idea. Let me explain: normalizing the data transforms it into a scale that is more intuitive. For example, we might say that we will normalize the data using the following rules:<br />
<ol>
<li>A value of 3 is good, so we'll call that 1</li>
<li>Any other value is bad, so we'll call that 0</li>
</ol>
<div>
Pretty cool. That's a pretty good one. Any time everything is ok, we would plot a 1. Any other status is undesirable and we would plot a 0. Transitions are still ugly and potentially troublesome. If we make one tweak, it could allow us to put some context around the resulting values. What if we changed it to 100% instead of 1 and 0% instead of 0? If we did that, we could actually put some additional meaning behind the resulting values. Thing about it, if everything is good, you're plotting 100%. What does that mean? It means that for 100% of the timeframe displayed, the status was good. If the status is 4, we'd see a line down at 0% meaning that for 100% of the timeframe displayed, the status was not good, or conversely: the status was good 0% of the timeframe. We could also create another data point which is the inverse of our normalized data:</div>
<div>
<ol>
<li>If value is 3, plot 100, else plot 0</li>
<li>Plot (100 - the value from above)</li>
</ol>
</div>
<div>
Doing this would also let us use a more intuitive graph called a stacked area graph. We could plot our normalized data using a pleasant color like blue or green and plot the inverse data using a warning color like red or orange. This would give us two series of data that compliment each other and when plotted as a stacked graph would look like the second graph <a href="https://docs.google.com/spreadsheets/d/1li0OORuHWwawmYxo7N0wnAT_jo9-BZboA3AM-s30xGE/edit?usp=sharing" target="_blank">here</a> (the first graph is a status code plot for reference):</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-gWUFIsW3MUo/XRuBWUp5rQI/AAAAAAABmGc/76Co40Asbu8knqGntpYFq970ZGk589QHgCLcBGAs/s1600/Screen%2BShot%2B2019-07-02%2Bat%2B11.07.37%2BAM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1526" data-original-width="1224" height="640" src="https://1.bp.blogspot.com/-gWUFIsW3MUo/XRuBWUp5rQI/AAAAAAABmGc/76Co40Asbu8knqGntpYFq970ZGk589QHgCLcBGAs/s640/Screen%2BShot%2B2019-07-02%2Bat%2B11.07.37%2BAM.png" width="512" /></a></div>
<div>
See how much more intuitive that is? The first graph in the above picture is a plot of the raw status code. How easy is it to know when things aren't good (without reading the axis labels)? Doable but not instantly intuitive. Imagine you had 12 of these kinds of graphs on a single dashboard on a wall. Would you want to take the time and effort to read the axis labels of each one to know how things are going? No. Now look at the second graph. It's pretty easy to tell that there were two different problems between 8pm and 11am and 1pm and 5pm. We could even remove the y-axis labels and you'd still probably be able to tell with a glance how things are doing. Imagine 12 different graphs like this. How easy is it to see if there's a problem? Just look for the red!</div>
<div>
<br /></div>
<div>
There are only two drawbacks. Have you noticed? We see that there are two problems, but are they the same problem? Actually, they're not. Also, are they the same severity of problem? The morning problem is that status goes from "subscribed" to "expired". The afternoon problem is that the status goes from "subscribed" (did you notice that it returned to a good status at noon?) to "trial". The morning problem is worse than the afternoon problem. There's a way to visualize this so that it all looks good. Let me explain:</div>
<div>
<br /></div>
<div>
Essentially we want to normalize the data but still keep as much detail as possible. We'll need to have four series, each with its own color. For any one data point, we'll only have a value of 100% in one of the series. All the others will be 0%. Meaning that for the timeframe that data point represents, whichever series has a value of 100% indicates the status for that moment. Let's look at the normalization rules:</div>
<div>
<ol>
<li>If the status code is 1, return 100, else return nothing (100 here means Trial status)</li>
<li>If the status code is 2, return 100, else return nothing (100 here means Unsubscribed status)</li>
<li>If the status code is 3, return 100, else return nothing (100 here means Subscribed status)</li>
<li>If the status code is 4, return 100, else return nothing (100 here means Expired status)</li>
</ol>
<div>
<a href="https://docs.google.com/spreadsheets/d/1li0OORuHWwawmYxo7N0wnAT_jo9-BZboA3AM-s30xGE/edit?usp=sharing" target="_blank">This</a> is what it would look like (graph on the right, first two shown for reference):</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-_0YfKQXguB4/XRuEM_BHFaI/AAAAAAABmGo/zWAAOYT-sRwq-MY_iPd6Vzz02x74j1ghgCLcBGAs/s1600/Screen%2BShot%2B2019-07-02%2Bat%2B11.16.47%2BAM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="992" data-original-width="1600" height="396" src="https://1.bp.blogspot.com/-_0YfKQXguB4/XRuEM_BHFaI/AAAAAAABmGo/zWAAOYT-sRwq-MY_iPd6Vzz02x74j1ghgCLcBGAs/s640/Screen%2BShot%2B2019-07-02%2Bat%2B11.16.47%2BAM.png" width="640" /></a></div>
<div>
Notice how the problem in the morning is highlighted with a red and the problem in the afternoon is highlighted with a yellow? Easy to tell that there are two problems, that they are different, and that the morning problem is the more severe. </div>
<div>
<br /></div>
<div>
This is what the final version would look like in the LogicMonitor web gui (not interesting I know since the status code didn't change the whole time I was building this):</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-EiDTSddWTbg/XRuGOh_7XPI/AAAAAAABmG0/XLoNgJoHu0I_bErzv3JkBRXnTHAtSL3hwCLcBGAs/s1600/Screen%2BShot%2B2019-07-02%2Bat%2B11.26.35%2BAM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="780" data-original-width="1288" height="386" src="https://1.bp.blogspot.com/-EiDTSddWTbg/XRuGOh_7XPI/AAAAAAABmG0/XLoNgJoHu0I_bErzv3JkBRXnTHAtSL3hwCLcBGAs/s640/Screen%2BShot%2B2019-07-02%2Bat%2B11.26.35%2BAM.png" width="640" /></a></div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Ex_8jJsfO-8/XWAPlWICqfI/AAAAAAABoWc/b10xZGwgUegZLRO4oO1q-MjZ1XPTAdNPACLcBGAs/s1600/Screen%2BShot%2B2019-08-23%2Bat%2B10.24.06%2BAM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="497" data-original-width="1600" height="196" src="https://1.bp.blogspot.com/-Ex_8jJsfO-8/XWAPlWICqfI/AAAAAAABoWc/b10xZGwgUegZLRO4oO1q-MjZ1XPTAdNPACLcBGAs/s640/Screen%2BShot%2B2019-08-23%2Bat%2B10.24.06%2BAM.png" width="640" /></a></div>
<br />
<br /></div>
<div>
Here's how it's built in the GUI:<br />
Notes on the screenshot:<br />
<br />
<ul>
<li>it shows line types of "Area" but they should be "Stacked" to display properly</li>
<li>the formulas should be `if(in(StatusCode,3),100,unkn())`</li>
</ul>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-YRHZQF8_UvE/XRuGqinOSOI/AAAAAAABmG8/5KdwXIfr35gofENP2MgtBiErPIX-kYHeACLcBGAs/s1600/Screen%2BShot%2B2019-07-02%2Bat%2B11.30.07%2BAM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1118" data-original-width="1600" height="447" src="https://1.bp.blogspot.com/-YRHZQF8_UvE/XRuGqinOSOI/AAAAAAABmG8/5KdwXIfr35gofENP2MgtBiErPIX-kYHeACLcBGAs/s640/Screen%2BShot%2B2019-07-02%2Bat%2B11.30.07%2BAM.png" width="640" /></a></div>
<div>
<br /></div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-41694115861847312302019-06-03T09:30:00.001-05:002019-06-03T09:30:21.032-05:00How to share a ton of stuff with someone else<br />
<h2>
If you have the stuff to share:</h2>
<ol>
<li>Download and install <a href="https://www.resilio.com/individuals-sync/?" target="_blank">Resilio Sync</a></li>
<li>After opening the app, click the plus sign in the top left corner and select "Standard Folder"</li>
<li>Browse to the folder you want to share with someone else and click "Open"</li>
<li>A new entry will appear in your list of folders. At the right end of this entry will appear three dots (when you mouseover the row). Click the three dots and select "Copy Read Only key" or "Copy Read & Write key" depending on whether or not you want the sharer to be able to change what's in the shared folder. They key is now in your clipboard.</li>
<li>Send the key to the person you want to share with.</li>
</ol>
<h2>
If you have received a key:</h2>
<br />
<div>
<ol>
<li>Download and install <a href="https://www.resilio.com/individuals-sync/?" target="_blank">Resilio Sync</a></li>
<li>After opening the app, click the plus sign in the top left corner and select "Enter key or link"</li>
<li>Paste in the key that was sent to you</li>
<li>Browse to the folder you want to synchronize and select Open.</li>
<li>Go grab a coffee and chips and wait until the synchronization finishes.</li>
</ol>
<div>
<br /></div>
</div>
<div>
<br /></div>
<div>
Disclaimer: using any protocol to transmit/receive data that you are not legally allowed to transmit/receive is obviously illegal. I'm not responsible if you use this to do something illegal.</div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-78929754463713958822019-04-16T09:01:00.001-05:002019-04-16T15:13:10.370-05:00One Trailer for Each Piece of the SagaSaw an article bringing together all the trailers for all the Star Wars productions. They did the episodes 1-9 first, then the ancillary productions. I decided to put them into <a href="https://www.youtube.com/watch?v=bD7bpG-zDJQ&list=PL0qIrrwWPNiJRSnE-NS1QglTRmBj0m_ji">a YouTube playlist</a>. You're welcome.<br />
<br />
Don't forget <a href="https://streamable.com/rgxvy">the one that you can't find on YouTube</a>, it's after Episode 6, before 7.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-33389234402512367462019-03-22T09:16:00.001-05:002019-03-26T11:31:47.242-05:00Web GL GlobeI have spent some time now working in the Oil Industry. I've been working in technology, so I haven't had much to do with actual oil. However, I did come across <a href="https://www.gsmlondon.ac.uk/global-oil-map/?ref=magazineduwebdesign">this really cool visualization of oil imports and exports</a> when I was searching for a way to visualize some network performance data. It's based on a bit of code by Google called the <a href="https://experiments.withgoogle.com/chrome/globe">WebGL Globe</a>. WebGL Globe is built on a technology called <a href="https://webglsamples.org/">WebGL</a>, which is like OpenGL except that it runs natively in the browser. It allows for interaction like what you would expect from a 3D graphics game but right in the browser with web code. Some of the <a href="https://webglsamples.org/fishtank/fishtank.html">examples</a> are pretty <a href="https://webglsamples.org/spacerocks/spacerocks.html">cool</a> and load extremely quickly because they don't really use the normal document structure. (Some other examples of really neat uses of WebGL and/or <a href="https://threejs.org/">three.js</a>, a library to facilitate using WebGL: <a href="http://stars.chromeexperiments.com/">Galactic Neighbors</a> <a href="https://beinternetawesome.withgoogle.com/en_us/interland">Internet Safety game for Kids by Google</a> <a href="http://www.larsberg.net/#/lubricious">Lubricious</a>)<br />
<br />
The concept is pretty expansive when you think about the kinds of data that can be shown. WebGL Globe combines data that has several dimensions of information encoded and visualized:<br />
<br />
<ol>
<li>Node Location - latitude and longitude</li>
<li>Node connections - showing that two nodes are connected in some way (shown as an arc when you click on a country in the World of Oil visualization).</li>
<li>Connection intensity - this one can have multiple dimensions depending on how creative you get. For example:</li>
<ol>
<li>the line color itself can indicate some sort of status (red/green/yellow/orange). It's even possible that you could show percentages of the arc length as certain colors to indicate distribution of the status (i.e. 10% is bad while 90% is good might mean a line that is mostly green with a segment of length 10% that is red).</li>
<li>the thickness of the arc can be another dimension, indicating something like volume</li>
<li>the maximum altitude of the arc can be another, indicating sample size</li>
</ol>
</ol>
<div>
All this is neat, but what good does it do anyone. Well, if you've watched my <a href="https://youtu.be/9vPZRViW43c">Analyzing TCP Application Performance</a> video, you know that monitoring of response time is the most important part of any infrastructure monitoring. If you're doing application response time monitoring, you should have data for each transaction showing how each transaction performed. That's a huge amount of data (big data anyone?). </div>
<div>
<br /></div>
<div>
Visualizing that data requires a few steps of summarization and grouping. First of all, every transaction's metrics should be retained individually so that you can dive into the details if needed. Second, you can start grouping by user then by summarizing by time buckets (1 minute or 5 minute). Another level of grouping that can be done is to group by network location. This can legitimately be done because most users at a particular location are going to share a very high percentage of the network path that gets them to the services they are consuming. </div>
<div>
<br /></div>
<div>
Let me rephrase: You should have data that describes sources, destinations, and the performance between them. Sound familiar? What I'm envisioning is taking this data and plotting it out using WebGL Globe. Each network location and each service hosting location is a node (hm, could cloud services actually be represented as clouds?!?). They'd be connected with arcs. Thickness of the arcs could represent the number of transactions between those two nodes. Height of the arc could represent the recent negative variability in performance or a way to highlight the selected arc. Color of the arc could show a measure of the deviance in performance. </div>
<div>
<br /></div>
<div>
Click on a node and you'd see the arcs from all the user locations consuming services from that site (if any) and the arcs to all the service hosting locations consumed by that site (if any) pop up in altitude (normally they'd be displayed at sea level or hidden based on a GUI setting). This is similar to the action you see in the Global Oil visualization when you click on a country (you see the countries connected to it). </div>
<div>
<br /></div>
<div>
From there, if you saw that all the arcs were showing problems, you would know (from problem domain isolation) that there is a problem common to all users at that site. You could click on the node again and get into performance stats for the infrastructure at that location (from a tool like LogicMonitor). Following the colors should get you to the root of the problem pretty quickly.</div>
<div>
<br /></div>
<div>
Alternatively, if you saw a problem in only one of the arcs (or a small selection) follow problem domain isolation tactics and click on the arc having the problem. That should dive you into the infrastructure connecting those two nodes so you can find the problem.</div>
<div>
<br /></div>
<div>
Nodes themselves can have problems that don't evidence themselves outside the node. If that's the case, you should be going into the node to look at why there are performance issues. You'd know to go to there because the node icon itself would be showing some color indicator that there's a problem.</div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-78888539123823946642019-03-21T20:15:00.000-05:002019-05-10T12:40:55.863-05:00RDP PingUPDATE: This content is no longer maintained here. Instead go to <a href="https://github.com/sweenig/RDPing">https://github.com/sweenig/RDPing</a>.<br />
<br />
<a name='more'></a><br /><br />
UPDATE: Version 2.3 Released! This version downloads TCPing if you don't have it. Since this script is now fully contained, I will no longer be hosting the script except as embedded in this post (<a href="http://pastebin.com/u/sweenig">thank you to my new code host pastebin.com</a>). <br />
<br />
UPDATE: Version 2.2 Released! This version fixes some bugs with the previous version and adds a help section. Run reboot.bat without any arguments and instead of rebooting the local box it will display help.<br />
UPDATE: Version 2.1 Released! This version adds a switch that allows you to skip the reconnect. Just add NOCONNECT after the name of the server.<br />
<br />
<center>
Original Post</center>
<hr />
UPDATE: I decided to just post this as a script.<br />
I also updated the script to just perform the whole reboot ping tcping mstsc series of commands. Now it only requires the name of the server to reboot (the credentials you used to launch this script will need admin access on the target server). It'll shut down the server, do a continuous ping, look for failed pings, then look for successful pings, then use <a href="http://www.elifulkerson.com/projects/tcping.php" target="_blank">tcping</a> to watch for the RDP service to come up. Once it does, it'll launch your RDP client and connect you to the server. Tcping will have to be in the same directory where this batch file exists. <span style="background-color: white;">I also noted a way of doing this for a bunch of servers at one time. Enjoy!</span><br />
<blockquote class="tr_bq">
<pre>@echo off
:: This script reboots a server, pings until it doesn't respond,
:: pings until it responds, then waits for RDP to come up, then
:: launches the RDP client and connects to the server. If your
:: password is saved and you don't have a welcome message before
:: logon, you should be brought directly to the desktop of the
:: server after rebooting.
:: the first and only argument is the name of the server to reboot.
:: use a command like the following to run this for a list of servers
:: FOR %A in (server1 server2 server3) DO (start reboot.bat %A)
::shutdown the server
echo.
echo %1 rebooting...
shutdown /r /d p:4:1 /m \\%1 /t 0 /c "Remote reboot requested"
if errorlevel 1 GOTO:EOF
::Ping until unsuccessful then successful
set pingfailyet=FALSE
set pingwaittime=3
echo.
echo Pinging %1...
:startping
ping -n 1 %1 | find "Reply"
if %errorlevel%==0 (
if %pingfailyet%==FALSE (
::echo %1 hasn't gone down yet. Pinging again in %pingwaittime% seconds...
CHOICE /C x /N /T %pingwaittime% /D x > NUL
goto startping
) else (
echo Successfully pinged %1.
goto endping
)
) else (
set pingfailyet==TRUE
echo No reply from %1.
CHOICE /C x /N /T %pingwaittime% /D x > NUL
goto startping
)
:endping
::use tcping to check when RDP becomes available
echo.
echo Waiting for RDP on %1 to become available...
tcping.exe -t -i %pingwaittime% -s %1 3389
if %errorlevel%==1 GOTO:EOF
::launch RDP
echo.
if NOT %2==NOCONNECT (
echo RDP is available on %1. Connecting...
start mstsc /v:%1 /f
) ELSE (
echo RDP is available on %1.
)</pre>
</blockquote>
<br />
<center>
Original Post</center>
<hr />
I find myself rebooting servers in remote locations from time to time. The standard procedure for checking to see whether or not the server is up is by doing a continuous ping. Once the ping starts timing out, you know the server has gone completely down. Once it starts responding, you know it's back up. Well, you know that the NIC is back online, which means the OS is online on some level. However, most of the time, the first thing I want to do when it comes back up is connect to the server via RDP. Most of the time, if you start trying to connect via RDP as soon as the server starts responding to pings, the RDP daemon isn't up and running yet. So you'll get a bunch of timeouts until a minute or two later when it finally comes online. So, I've wanted a way to check to see that the RDP daemon is up and running before trying to connect.<br />
<br />
Now I have a way. There is a great little utility called <a href="http://www.elifulkerson.com/projects/tcping.php">tcping.exe</a> that does what ping does, except it does it for TCP ports instead of just checking the IP address. However, in order to not have to set things up every time and also to automatically kick off the RDP session, I've put together the following batch file:
<br />
<div>
This can be run either from a shortcut (tip, for Windows 7 users put it in the All Programs folder and you can run it from the start menu just by typing rdping) or it can be run from the command line with the name or IP address as the only argument of the batch file.</div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-9129463197233057363.post-69484789032291323392019-03-08T14:03:00.001-06:002019-03-08T14:03:06.080-06:00Using DockerJust a couple articles that have been camping out in my browser since I found them. They were very useful in helping me get docker doing useful stuff, like <a href="https://github.com/sweenig/docker-ansible-playbook">Ansible</a>.<br />
<br />
<a href="https://blog.docker.com/2016/09/build-your-first-docker-windows-server-container/">https://blog.docker.com/2016/09/build-your-first-docker-windows-server-container/</a><br />
<br />
<a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04">https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04</a><br />
<br />
<pre>alpine/git latest a1d22e4b51ad 10 days ago 27.5MB
ansible-docker latest 25b39c3ffd15 2 weeks ago 153MB
ubuntu latest 47b19964fb50 4 weeks ago 88.1MB</pre>
<br />
Ansible-docker is a container I modified to suit my needs. You can use it by either cloning the git repository and building from source or just pull it from the hub:<br />
docker pull sweenig/ansible-docker<br />
<br />
I used this to know how to push images to docker: <a href="https://ropenscilabs.github.io/r-docker-tutorial/04-Dockerhub.html">https://ropenscilabs.github.io/r-docker-tutorial/04-Dockerhub.html</a><br />
<br />
alpine/git is the easiest way I've found to run git on Windows (hint, it's not actually running in Windows but in Linux inside the container).Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-65866175521595594232019-03-07T10:03:00.000-06:002019-03-07T10:03:10.689-06:00Object Oriented ProgrammingToday's blog post may have been posted before, but it's a really good one. If you are looking to get into object oriented programming, you should give this a look: <a href="https://inventwithpython.com/blog/2014/12/02/why-is-object-oriented-programming-useful-with-a-role-playing-game-example/">https://inventwithpython.com/blog/2014/12/02/why-is-object-oriented-programming-useful-with-a-role-playing-game-example/</a>.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-2819803366544506512018-05-08T08:36:00.001-05:002018-05-08T08:36:07.593-05:00HTML Maps, ContinuedContinuing <a href="http://stuart.weenig.com/2017/12/html-maps.html">a previous post</a>, I decided to add some CSS to make it obvious that you can click on the mapped links. Here's the CSS:<br />
<br />
<pre><style>
a:hover {border:1px dotted gray;}
a {position:absolute;}
</style>
</pre>
<br />
Obviously, you may want to use <a href="https://www.w3schools.com/cssref/css_selectors.asp">CSS selectors</a> to make sure that only your mapped links on images get styled this way.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-10933915839143706492017-12-13T08:30:00.000-06:002017-12-13T08:30:20.756-06:00Online circuit schematic design and simulationThis is pretty cool, although I haven't actually had a chance to use it since most of my circuits don't involve much signal processing and are quite simple DC circuits. However, <a href="http://lushprojects.com/">LushProjects</a> has this <a href="http://lushprojects.com/circuitjs/circuitjs.html">circuit simulator</a> that is completely online and free to use (unlike SPICE). The neat thing is that you should be able to embed your own circuit onto any web page using this tool and iframe.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-43752611155113061832017-12-12T13:39:00.000-06:002017-12-12T13:39:08.478-06:00Hard puzzle with an easy solutionI've been keeping <a href="http://datagenetics.com/blog/december32015/index.html">this tab</a> open on my browser because I wanted to work out the solution myself. I finally figured it out (2 years later). Don't give in and look at the solution without giving it a good try. Hint, you can do it without any trigonometric functions and without Pythagoras' help.<br />
<br />
This is known as <a href="https://en.wikipedia.org/wiki/Langley%E2%80%99s_Adventitious_Angles">Langley's Adventitious Angles</a>. And a good visual solution can be seen <a href="http://agutie.homestead.com/files/LangleyProblem.html">here (warning Flash required)</a>.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-3749348819234481712017-12-11T11:01:00.000-06:002017-12-11T11:01:16.468-06:00Boolean ArithmeticI had to explain Boolean arithmetic the other day to non-makers. These guys didn't have any real experience with electric logic circuits, but <a href="https://www.allaboutcircuits.com/textbook/digital/chpt-7/boolean-arithmetic/">the pictures here</a> seemed to help.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-7978867543927767232017-12-08T10:47:00.000-06:002017-12-08T10:47:29.681-06:00Prusa's ColorPrint toolWhen 3D printing, there's usually a jump in cost and complexity for printers that can print multiple colors. As a workaround, you can pause the printer, switch the filament to a different color, print a few layers with that different color, then pause, switch filament, then resume printing. This can be a very tricky thing to do manually, so obviously, there is <a href="https://www.prusaprinters.org/color-print/">a tool to do it</a>. Here are <a href="http://infinity3dprints.com/product/custom-ear-rings/">some prints</a> by a buddy of mine that utilized this tool for these prints. He prints a black surface with a cutout for the image, prints a few layers of a different color (glow in the dark in this case), then resumes printing in black. He didn't have to design the model with any major modifications, just the first few layers cut out so that the glow in the dark layer can shine through.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-69404284503038082282017-12-07T10:44:00.000-06:002017-12-07T10:44:47.367-06:00PiVPNRolling your on VPN can have various benefits. The biggest of which is that when you're on an unsecured network (i.e. any wifi network that you don't own yourself) your traffic is encrypted back to your home and then goes out to the internet. This means that you don't have to <b><i>trust </i></b>that the WiFi owner (think Starbucks or McDonald's) isn't snooping on your packets. It doesn't matter if they do snoop it because you're packets are encrypted and nobody can understand it unless they are you or your RaspberryPi at home.<br />
<br />
Before you contest, yes, I know that any form of encryption can eventually be beaten. If you're that paranoid about someone decrypting your packets (which would take years by the way) you should be off the grid.<br />
<br />
That said, I looked into setting up a VPN option for myself and eventually found <a href="http://www.pivpn.io/">PiVPN</a>. This little one line installer sets everything up on your RPi so that it becomes a VPN endpoint. Use it to generate a certificate which you can load on your device (I've tested on iOS and Windows 10) into the freely available <a href="https://openvpn.net/index.php/open-source/downloads.html">OpenVPN client</a>.<br />
<br />
I have since found, but not installed/tried <a href="https://github.com/mitchellurgero/pivpn-gui">a web GUI</a> that should let me manage PiVPN through a browser. I hope to try this eventually after I have some free time. So, probably next year!Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-13222926608205352232017-12-06T10:08:00.000-06:002018-09-10T12:01:25.245-05:00Sense HAT Data Logger, my versionTHIS CONTENT HAS MOVED TO GITHUB: <a href="https://github.com/sweenig/SenseHatDataLogger">https://github.com/sweenig/SenseHatDataLogger</a><br />
<br />
<a name='more'></a><br />
I ran across <a href="https://projects.raspberrypi.org/en/projects/sense-hat-data-logger">this project</a> and decided to give it a try. Without actually trying it out on a physical SenseHAT, I simplified the code down to this:
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-18869114546704953332017-12-05T10:00:00.000-06:002017-12-05T10:00:09.592-06:00PressPiI wanted to figure out the best way to put WordPress on a RaspberryPi. Turns out the best way is an image called <a href="http://everyday-tech.com/wordpresspi-wordpress-web-server-image-on-raspberry-pi/">PressPi</a>. Load this up, lock down all the security, load CertBot so it's all running over https and you're good to go.<br />
<br />
In case you're interested, here's a good set of instructions for <a href="https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-lamp-on-ubuntu-16-04">installing Wordpress on Ubuntu 16.04</a>. So many instructions! You might need <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-phpmyadmin-on-ubuntu-16-04">phpMyAdmin</a> as well.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-33043932460176071362017-12-04T09:55:00.000-06:002017-12-04T09:55:18.225-06:00HTML MapsI recently needed to overlay a bunch of links on top of an image. This is done one of two ways, CSS being the more modern way. Essentially, you create a div with a bunch of elements inside it, which elements are all positioned absolutely.<br />
<br />
<div style="position:relative; height:786px; width:537px; background:url(myimage.png) 0 0 no-repeat;"><br />
<a style="position:absolute; top:393px; left:147px; width:87px; height:69px;" title="asdf" alt="asdf" href="asdf" target="_self"></a><br />
</div><br />
<br />
Instead of mapping out all the positions manually, there's a <a href="http://www.html-map.com/">really neat tool</a> that will let you do it right on top of your own image and then generate both the HTML map code as well as the more simple CSS code to render it on a webpage.<br />
<br />
<br />Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-81232084038429765272017-12-01T09:54:00.000-06:002017-12-04T08:24:24.251-06:00W3Schools and their incredible CSS LibraryAnyone who has built a website in recent history knows the importance of good CSS. I myself have been compiling a master CSS sheet that I use on most of the web development projects I'm involved with. I realized that I was trying to accomplish the same set of outcomes over and over, so a standard library of CSS styles was a natural shortcut to a good end.<br />
<br />
I know some have been critical of <a href="http://w3schools.com/">W3Schools.com</a> and their no nonsense way of explaining web development concepts, citing technical inaccuracies and nuances. I've found that those perceived inadequacies either can't be discerned by "normal" people or don't have a discernible impact on the end product. As such, I've been a fan for a couple years now. I've built their site into my Google searches so that I know I'll end up going to the answer that I'm sure they've provided straightaway.<br />
<br />
I've used a few of their tools from the CSS section over the last few years, particularly pleased with their tooltip implementation. That's when I discovered that the CSS sheet that they use for their own site, which has <i style="font-weight: bold;">all</i> of the CSS needed to implement all of the cool, modern utilities is <a href="https://www.w3schools.com/w3css/">free to use</a>. They even encourage it!<br />
<br />
There are a couple things I like about it:<br />
<br />
<ol>
<li>All their examples use this single sheet. I don't have to understand a concept, then look up a different place to find out how to use the W3.CSS framework to implement it. </li>
<li>It uses pure CSS. I only include one CSS reference and I'm good to go. There's no need to import a jQuery/javascript library as well to make it all work.</li>
<li>It treats responsiveness and mobile first as the highest priorities. This is what makes simple websites look like websites developed my multi-billion dollar corporations.</li>
<li><a href="https://www.w3schools.com/w3css/w3css_templates.asp">Templates</a>!</li>
</ol>
<div>
I used one of the templates <a href="http://resume.weenig.com/">here</a>. It's one page. I don't host any javascript nor CSS files. Even the icons are used from frameworks <a href="https://www.w3schools.com/icons/default.asp">referenced and explained by W3Schools</a>.</div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-4508192869292499752017-11-30T09:14:00.000-06:002017-11-30T10:05:49.464-06:00Installing LAMP in one stepGo educate yourself on <a href="https://en.wikipedia.org/wiki/LAMP_(software_bundle)">LAMP</a>.<br />
<br />
Installing LAMP has been getting easier over the years. Now you can install it with a single command line:<br />
<br />
sudo apt-get install lamp-server^<br />
<br />
More information <a href="http://www.sudo-juice.com/install-lamp-server-ubuntu/">here</a>.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-1890027225836332492017-11-30T09:10:00.000-06:002017-11-30T10:05:49.405-06:00Encryption EverywhereAnybody who has stood up a web server knows the importance of securing that connection. Watch this video:<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/I3a0bItZ3sI?rel=0" width="560"></iframe><br />
<br />
While I don't yet use the HTTPS Everywhere add-on, I do make use of <a href="https://certbot.eff.org/">Certbot</a>. You can see an example <a href="http://resume.weenig.com/">here</a>. This website runs on a LAMP server on AWS (the free tier). From beginning to end, except the coding of the site itself, I had the secured site running in about 15 minutes. Several cool things happen when using Certbot:<br />
<br />
<ol>
<li>It's aware of the multiple hosts you may have configured in your web server and lets you run for specific hosts.</li>
<li>It automatically configures http redirect. This means that even if a user accidentally left of the https:// from the address to your site, they'll get redirected to the https version automatically. When I first did this manually, it took me several days to get it working right. </li>
<li>The certificates are free because they have a short life span. So, Certbot has to be run regularly to get a new certificate. You don't have to pay attention to that cycle though because you can run the checker daily or weekly and it won't do anything unless the existing certificate is close to expiration.</li>
</ol>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-265922875055179332017-11-30T08:34:00.003-06:002018-05-08T08:25:13.615-05:00Mini blog postsSince I don't really have the time anymore to do long, in-depth blog posts, I've decided that I'll start doing mini posts with tidbits of information. When I started this blog, it started out as a place for me to post stuff I needed to remember and/or have a place to write stuff down that I could recall easily. This is a continuation of those efforts. I'll be picking suspended Chrome tabs and detailing why I've kept that particular tab around.Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0tag:blogger.com,1999:blog-9129463197233057363.post-67206417349265630532017-09-12T18:30:00.001-05:002017-09-12T18:30:35.666-05:00Hurricane Harvey - Our Story<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Hurricane Harvey started affecting us Friday, 25 AUG 2017. It was my Friday off, and we were preparing for the next Monday when the twins would enter kindergarten. As such, Friday was "Meet the Teacher" at their school. The sky was dark and there was light rain. It felt like a good day to cuddle up and watch a movie. Friday evening, I was in touch with our ward leadership as we coordinated our response teams. </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our band is to perform on 9 SEP, so the next morning (Sat 26 AUG), Christy and I got together with the rest of the band and had practice over near Black Horse Ranch. We were finishing up with the first set as the bottom fell out so we decided to break. We were across Cypress creek from our kids and we wanted to make sure we got back to them before any flooding started. We had identified some friends of ours who were in a neighborhood near us that was likely to flood. They were preparing to move out the following Thursday (31 AUG) so they had most of their stuff in boxes already. She (Kelia) is almost 9 months pregnant so it was agreed that they would preemptively evacuate to our home. On the way home, we went over and helped him (Kris) lift some of their furniture onto blocks and 2x4's. They had a few other things to finish and he has a large pickup truck, so we left them to finish expecting them to come over later in the day. They could get out even if the flooding started. It's important to note that their neighborhood usually floods before anything else in the area and when the flooding is really bad, their neighborhood dumps out into our neighborhood. As long as they are not flooding into us, we don't have a problem draining our neighborhood. </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They came over later that afternoon and the kids started playing. We got them setup in our spare bedroom with bunk bed cots for the kids (4 total people in that family). Kris and I went back over to his neighborhood. He went back over to the house to get a few things they had forgotten and I went to help another friend attempt to waterproof his garage door. We jammed some tarps into the hinges of the garage door and weighted it down with landscaping bricks. This turned out to be a pretty good barrier against the water that eventually rose a couple feet above the bottom of the garage door. When we returned, there was only water in the street gutters. A few hours later, water had risen to cover the street. We started keeping an eye on things. The reservoir we drain into was well below us, so I wasn't worried that flooding would get to dangerous levels for us. The last two major floods had not produced enough water quickly enough to have it come up more than halfway up the driveway.</span></div>
<b id="docs-internal-guid-34127d44-7869-01c0-c493-aa5110bd4a48" style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sunday morning (27 AUG) dawned with some water covering the streets, but less than the highest point overnight. The other friend in the same neighborhood that always floods first had not yet evacuated. The overnight rise of water had not receded so they were looking at evacuation options. Kris and I rode in his big truck and started to prepare their house for flooding and to convince them to evacuate. It became evident that the water was going to keep rising. Last year, this family had waited until it was too late to evacuate during the 2016 tax day flood and had to be evacuated by canoe. We emphasized how important it was to avoid getting to that point again. The father wanted to wait it out, so we took the mother and kids to another ward member's two story home which was serving as a dispatch location for the emergency crews. The father was left to his own devices to get out (which he eventually did on his own).</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A large number of ward members had congregated at that two story home for a previously scheduled baptism. Since extended family had flown in for the event, it was decided that the baptism would be performed not at the presently closed Eldridge building, but in the pool in the rain (pretty memorable!). Since there was a large group and the Bishop was present, it was decided that the sacrament would be administered since all other church meetings were cancelled. Shortly afterward, the rain lightened up and most of our street drained.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sunday afternoon, a rescue request came in for a family in Enchanted Valley. It was outside our area of responsibility, so our dispatch tried to find resources in the area. Unfruitful, he dispatched Scott with his big Yukon. They made it to the family and loaded everyone up. There wasn’t room for two of the four rescuers, so they stayed behind to be retrieved after dropping off the family at a safe location. Upon returning, Scott decided to splash around a little and stalled his Yukon. They pushed it up onto dry land and notified our dispatch. I saw that call come in and reached out to a few Jeeper groups who had been offering help and making rescues since the rain started. Allan and Z responded and we headed out toward Telge and 290. I made it past the Sheriff station before the water started getting deeper. I gave Allan and Z my tow strap and they continued on (they have a few inches more clearance than I do). Their two door jeep would only hold two of the four rescuers, so Scott and his nephew stayed with his truck while the two who had originally stayed behind were brought back to me. I had backtracked and waited under the 290 bridge at Telge. Upon returning, the water had risen. Allan commented that he could probably make it back in, but he was worried about getting out while the water was still rising. We decided to attempt to rescue via Huffmeister. It was dark by this time. In my lower Jeep, I led the charge. The streets were clear of water until about a half mile north of Cypress North Houston. I was cruising at about 40mph when we hit the water. Needless to say, it was a pucker moment. We were all fine, but it was one of those moments where everything went into slow motion. The water started getting deeper, so again Allan in his swamp thing went ahead to see what they could see. The water ended up being too deep (reportedly about 6’) so, we weren’t getting to Scott and his nephew any time soon. They would have to ride it out. The rain started coming down harder and we had just received news that the Addicks and Barker reservoirs would be opened up at 2am. Not yet knowing how this would affect the current water levels, we decided to break for the night. We went to bed late Sunday night as we watched the waters begin to slowly creep over the street in front of our house.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Monday morning (28 AUG) when Christy got up, Kelia told her the water outside, which was up to the sidewalk, was no longer draining away. Christy insisted that we start making plans to raise our important possessions and make an evacuation plan. I was hesitant because of past experience with extreme flooding had never given us any problems. I reluctantly conceded though and we figured out what we would do if we decided to evacuate. </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We found out that the neighborhood that always floods first had breached the main road and was spilling into our neighborhood. It wasn't going to get any worse for them, but it was coming in quickly enough that our drainage system wouldn't be able to keep up for long. I went for a hike in my chest waders and saw our main drainage creek rising. This meant that what we were draining into was full and it was only going to get worse from here. This had never happened before. It turns out that the Addicks reservoir had filled up. <a href="https://www.usatoday.com/story/news/nation-now/2017/08/28/controlled-release-water-houston-reservoirs/607594001/">The Army corps of engineers had already opened it up to drain it</a> (which would send the water south toward the ocean) but the water leaving was less than what was coming in. I broadcasted my hike live over Facebook (</span><a href="https://youtu.be/-XQDWJn-1Tw" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">https://youtu.be/-XQDWJn-1Tw</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). Upon returning home, I had decided it was time to get out while the water was low enough for my Jeep and Kris' truck. Our neighbors (4 adults and one infant) also needed to evacuate. We rallied everyone into motion and started implementing our plan. We got everything that we could think of up a as high as we could. Kris and I loaded our vehicles with the essentials we would be taking with us. I got my family loaded up in my Jeep and Kris got his and the neighbors loaded up in his truck. My neighbor got 3 videos of our escape (</span><a href="https://www.facebook.com/steven.dewey.12/videos/10212407896810647/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">part 1</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="https://www.facebook.com/steven.dewey.12/videos/10212407833449063/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">part 2</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="https://www.facebook.com/steven.dewey.12/videos/10212407861849773/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">part 3</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) from the back of Kris’ truck.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">My Jeep dove into the water and got us to a high point right before the exit of the neighborhood onto Barker Cypress (which was the spillover point for the first neighborhood that flooded). I parked there and we made sure the boys had their life jackets on and seatbelts off (in case we had to ditch the Jeep). About 8 minutes later (which seemed like an eternity) Kris' truck caught up with us and we pushed forward into the deeper water right before the exit onto Barker Cypress. It's at this point that I think I got water in my differential, more on that later. With no option but to push forward, we got water up to top of the Jeep tires before making it out onto the shallows of Barker Cypress. We turned south away from Cypress Creek and away from 290 where the water was coming from. We made it down to Tuckerton without any real issues except for some water up to the middle of the Jeep tires. It was dry from there on out. My plan was to head Southwest until we found a place to land. While en route, James, a fellow Cub Master, texted me offering to let us come to his place indefinitely, which we did. His neighborhood was wet but didn't have any water on the streets. I realized later that we were living out the story of the three little pigs, Kris' family fleeing the straw house from the big bad Harvey to our house of sticks, which we eventually fled to James' house of brick.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I spent most of Monday afternoon coordinating rescues, surveying potentially flooded/closed streets, and making various runs to the Longenbaugh Mormon church, which had been turned into a shelter. There was a ton of food and other donations to be received and sorted as well as families to take care of. I got a call later in the day asking to help with evacuation of a family of 13 near </span><a href="https://goo.gl/maps/JqjZtViGWK12" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">the intersection of Queenston and Tuckerton</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. I made it to the Shell station there, which had turned into a staging point for various high clearance vehicles that were going in to make rescues. I arranged for an ATV with a flatbed trailer to make the run into the house where the family was. They would bring them out to me and I would take them to a shelter. After the ATV was dispatched, they family called and cancelled. I still feel bad for the driver of the ATV. I got signed up to do a shift on Tuesday from 4-8pm at the Longenbaugh building. The shelter required two Elders or High Priests present at all times.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tuesday morning (29 AUG), Christy and I ventured out in the Jeep to try to make it back to the house. Several reports on our neighborhood Facebook page indicated that the waters were receding. We found high water on Red Rugosa, but not too much covering the street in front of our house. We discovered that the water seems to have entered the garage and gotten to the front porch, but didn't come into the house. We also found </span><a href="https://www.facebook.com/photo.php?fbid=10101621886254238&set=a.557776669418.2093987.49701026&type=3" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">a telephone/electrical pole</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which had been parked at the end of the street waiting to be installed that had </span><a href="https://www.facebook.com/sweenig/videos/10101622073508978/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">floated into our front yard</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This was surprising, but not unreasonable. It was wood and the water was high and apparently thrashing towards the three storm drains in front of our house. Some neighbors saw it churning around and lashed it to one of my trees. Christy and I elevated a few more things in case the water came up some more. I built a couple of impromptu sandbags out of wet towels, garbage bags, and landscaping bricks to put at the front and back doors. I spent the afternoon at the Longenbaugh building. The clouds moved off, the sun shone. It felt like a good sign that the storm was over. There was a rumor about a kicked in door across Barker Cypress, so I decided that I would spend the night at the house with my shotgun. It also gave me a chance to watch Guardians of the Galaxy 2.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Wednesday morning (30 AUG), </span><a href="https://www.facebook.com/sweenig/videos/10101623223419548/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">our roads were dry</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and we had decided that we could probably come back to our house from James'. The sun had started to shine and only the lowest intersections still had water. Harvey had moved on to east Houston, so while we were technically still in the storm, we were now on the dry side. We came back home and started to put things back down on the ground. I spent the afternoon loading up small items from Kris’ house and being a shuttle driver for the ward team that was gutting a home. I eventually got some food brought in for the teams in both locations.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thursday (1 SEP) we spent most of the day moving Kris and his family into their new home. The roads were dry, so it was a simple matter of loading up the U-Haul twice. Their new home is less than a mile away, but on our side of Barker Cypress (less chance of flooding).</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Friday (2 SEP) I spent the morning getting some things back in place around the house until my brother, John, came in from Dallas. When he got here, we got together with the ward team to work on removing some wood flooring from a flooded house. That took the rest of the night and we only got the main living room (150/1200 sq. ft.).</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Saturday (3 SEP) I dropped my brother off with the team that would continue for the next six hours working on that wood floor. I had arranged to attend a differential fluid changing party hosted by a shop owner on Clay road just inside the beltway. They were changing fluids for free, so it was a good opportunity to make sure everything was in working order and also make sure I got the water out of my gears. They also gave me some pointers which made installing the wiring harness for my trailer hitch dead simple. I got done with that around 1pm and went to act as a coordinator for the team that was finishing the wood floor removal and the other team that had begun gutting another house. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sunday (4 SEP) began with gutting a few houses in our neighborhood. We had abbreviated church meetings at 1pm during which a new Bishopric was called and we were notified that we would be meeting back in the West road building for the foreseeable future. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Monday (5 SEP) was Labor day and our crew chief had advised that those of us who had been working for several days straight take some time off to recover. I heeded that advice and played with the kids. I brought out the slot car track and </span><a href="https://www.facebook.com/sweenig/posts/10101630206051298" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">we raced</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In the evening, grandpa invited us over to go fishing. He had just bought three new kids fishing poles (Star Wars themed, of course). Luke caught a baby brim and a baby bass. I caught a turtle and Grandpa caught a brim and another turtle. We let the first one go, but decided to relocate the second one since the turtles have a tendency to kill the ducklings. Cole became an expert caster, sometimes throwing his practice weight 25 feet from the shore. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tuesday (6 SEP) meant a return to work; the Chevron offices had been closed since the storm. It appears the tunnels were flooded since the demo work had already been done and there were </span><a href="https://www.facebook.com/sweenig/posts/10101631348007808" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">dozens of fans and dehumidifiers</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
Stuart Weenighttp://www.blogger.com/profile/08727259847102590099noreply@blogger.com0