Wyndham and Miller Marketing

I am a huge fan of David Thorne’s 27bslash6 website, not the least because he references the movie Brazil.

Fans of his site are aware of his on going feud with office-mate Simon, and he took it to a new level this week by defacing Simon’s page on the the corporate website of Wyndham and Miller, where David works.

David announced in a tweet that the defacement had been up for 26 days without anyone noticing. It’s brilliant, and I was excited because now that I know where David works perhaps I could hire him to do some marketing for OpenNMS, assuming he still worked there.

However, as hours then days went by without the site being corrected, I got suspicious. So I did a search on “Wyndham and Miller” and found no references outside of the website. Odd for a marketing company founded in 1996 to have no web presence. Then I did a “whois” and noticed the domain was registered just this week.

Brilliant.

The site is beautiful. Any small marketing firm would be happy to have such a site, and under David’s profile is a very good summary of his abilities.

Too bad they don’t exist. (grin)

Mike Doughty, Ubuntu 12.10 and Amazon

On Sunday I got back in the country after a two week holiday in the South Pacific. It was an awesome trip: no Internet and, for most of the time, no shoes.

I got my brain regrooved.

Now I’m trying to dig out from under the backlog, and I noticed that Mike Doughty has a new album called “The Flip is Another Honey“.

I’m a huge Mike Doughty fan, and since I’m always eager to listen to his work (even an album of covers) I went to Amazon and bought the digital version.

Now, I use Ubuntu as my desktop O/S, and while I still run the “Long Term Support” 12.04 release at home and on my laptop, I recently upgraded to 12.10 at work to see if the MTP support was any better (it isn’t).

One of the more controversial changes in 12.10 was the addition of an Amazon shopping “lens” to the Unity desktop that would return Amazon search listings as well as local (to the machine) results. It’s pretty easy to disable, but I must admit it is a little annoying.

When I access amazon.com via Firefox, an Amazon icon shows up in the launcher as if I’d launched an Amazon application. I really don’t need another icon in my launcher, especially one that duplicates functionality I already have in my web browser. In fact, that’s kinda what web browsers were for: getting rid of lots of little “apps” and just having a single interface to remote content.

(sigh)

I know Canonical is doing this for the Benjamins, and considering the amount of money I’ve paid to them (i.e. zero) I really can’t complain, but it rubs me the wrong way, much like the default Samsung software on my Galaxy S3 that considers the Yellow Pages search application a “system” app that can’t be removed. I’m a little more upset about that, because I did pay money for my phone, which is why I run Cyanogenmod, and I hate additional kruft of all sorts on my machines.

But the main thing that bothers me is that even with this new “integration” I still can’t download the music I buy on Amazon in one step. For “The Flip” I had to download each of the 15 songs individually since Amazon requires the “Amazon Downloader” to manage its .amz files. I looked to see if there was support for the Amazon Downloader in 12.10 but couldn’t find anything.

I would think that considering how much Linux-based software Amazon uses internally and the fact that Ubuntu went forward with this integration despite the potential to piss of their users, the least they could have done is create an Ubuntu Amazon Downloader client.

I look to Canonical to drive a lot of Linux desktop support (see Valve’s decision to provide their initial Steam Linux client on Ubuntu) and I can only hope that we’ll see better execution in the future.

Juniper SRX240 Unboxing

Since the Apple fanboys seem to like unboxing their gear, I thought it would be fun to do the same with a new Juniper SRX240 I just bought (from Redapt – our equipment vendor of choice).

Due to my previously detailed issues with Centurylink, we are working with Time Warner to get a dedicated fiber circuit to the building. While the 10Mbps down will (in theory) be the same as what we had with the DSL line, it’s the 10Mbps up that I’m looking forward to the most.

Currently, we use a Cisco 800 series router to terminate the DSL line, and I needed something else for the new circuit. I’ve soured on Cisco in the last few years (due more to issues with their hiring practices than anything to do with their product) and I’ve grown real fond of Juniper gear, not the least because they are a customer (I buy my pizza at Papa Johns, I book travel on Travelocity, and I shop at Sears in part because I like being a customer of my customers).

Anyway, it looked like the SRX240 would be both more than sufficient for what I needed and it comes with lots of cool features.

I liked the minimalist packaging. The router came in a plastic bag held in the box with lightweight, molded plastic spacers – no “peanuts”. There was another bag with some documentation and a console cable, and the only other thing in the box was a power cable.

The router itself is rather lightweight. It’s 1U in height and about half the depth of a standard rack.

Since the circuit won’t be in for a couple of weeks there isn’t much configuration to do at the moment, but I was able to get the management interface configured from the command line which gives me access to the web-based user interface.

I tend to be a command line person, but when learning new gear having a webUI will be helpful.

Paulo, a Juniper employee who happened to be in our office when this arrived, showed me a neat trick with the CLI.

When you make a change to any router configuration, you usually have to “commit” it in order for it to be applied. This can be a bit scary, especially if you are remote or if you depend on the router for network access. One mistake and you might end up in the car.

JunOS has a feature called “commit confirmed”. This will commit your changes, but only for ten minutes unless you issue a normal “commit”. Thus if you screwed something up, ten minutes later the changes will be rolled back.

Pretty cool.

Politics

The following is another of my navel-gazing posts that have nothing to do with OpenNMS. Please, as always, feel free to ignore.

Tomorrow in the United States we begin the final process to electing a new President. I hope that by late tomorrow evening we have a decision, but with the fact that presidential elections have been so close in the last decade means that there is a small but significant chance that we won’t know the outcome for some time.

Ever wonder why elections have been so hard to call since 2000 or so? My theory is that there are really no significant differences between the two major candidates.

I think this started with Bill Clinton. With the success of Ronald Reagan’s campaigns, Clinton realized that he needed to co-opt some of that populist message. Thus started a blending that Presidential candidates use to try to appeal to as wide an audience as possible, resulting in a blurring of their differences.

Now many of you reading this will say that I’m full of it, and that Romney and Obama are totally different. I disagree. As the figureheads of the Republican and Democratic parties they share the same agenda: consolidate power into the hands of the few and use that power to control the rest of the populace.

I was trying to sum up my feelings in a few words, but Conor Friedersdorf in this column in The Atlantic summed it up nicely:

Neither the Democratic nor the Republican candidate in this race is trustworthy or desirable as a leader. Obama is a left-leaning technocrat who habitually breaks his promises and is eager to assume near dictatorial powers in the realm of national security. He has little regard for the Constitution or the recklessness of the precedents that he’s setting. And Romney? He’s a right-leaning technocrat who unapologetically breaks his promises, is eager to assume near-dictatorial national-security powers, and has little regard for the Constitution.

Now I happily voted for Obama in 2008. I was tempted to vote for him again, mainly because the Republican attacks on him were, if not outright lies, quite often wrong. Take spending for example. Obama is attacked for “spending our children’s future” yet the growth of government spending under his administration “has actually been trivial compared to the last 4 presidents.” His administration is attacked for health care reform. As a small business owner who provides health care insurance for our employees I’ve seen my rates double – twice – in eight years. Something needs to be done. While the plan that passed is far from perfect, the idea of just repealing it and going back to the status quo is appalling. Finally, Obama is criticized for not creating enough jobs, but the Republican controlled house refused to implement any of his ideas by rejecting every jobs bill he proposed. It’s okay to disagree, but you better have a solution of your own if you want credit from me. It’s easy to just say “no” to everything. Considering the conditions he inherited and Congress, I think Obama has done an amazing job with the economy.

No, where Obama fails is in the area that is nearest and dearest to my heart: civil liberties. My view is that government should provide a level playing field and then get out of the way. More and more I’ve seen government taking an active hand in trying to control the populace. 9/11 has been used to reinforce the idea that people should do two things: live their lives in fear and buy stuff. When dealing with a person who is inconvenient to handle as either a criminal, subject to the rule of law, or a prisoner of war, subject to rules such as the Geneva Convention, why not create a new category called “enemy combatant” subject to no rules? Guantanamo is still open, the Patriot Act was renewed and strengthened, drones are being used to spy on American citizens and the NSA is building a huge data center to do the same on the Internet. And that’s just a small part of it.

So if Obama hasn’t earned my vote, that means it should go to Romney? No. I don’t see Romney doing anything better and he would probably make things worse, faster.

This year I am voting for Gary Johnson. While he has zero chance of winning, he is the only third party candidate on the North Carolina ballot, and while I rarely identify with the “Big L” Libertarian party, I like Johnson. It would be nice if his voice could have been heard in this election.

What’s most important to create viable third parties is a candidate to receive matching funds for their campaigns. In order to do so, they must get at least five percent of the vote:

Minor party candidates and new party candidates may become eligible for partial public funding of their general election campaigns. (A minor party candidate is the nominee of a party whose candidate received between 5 and 25 percent of the total popular vote in the preceding Presidential election. A new party candidate is the nominee of a party that is neither a major party nor a minor party.) The amount of public funding to which a minor party candidate is entitled is based on the ratio of the party’s popular vote in the preceding Presidential election to the average popular vote of the two major party candidates in that election. A new party candidate receives partial public funding after the election if he/she receives 5 percent or more of the vote.

I think the only way out of the quagmire created by the Republicrats and the Demmicans is to have several strong parties so that rule has to be more of a compromise instead of just “an endless slap fight” between two of them.

Still, several people have accused me of throwing my vote away, especially since the race is so close in North Carolina. That argument was almost enough to sway me, especially when thinking of things like the Supreme Court (where I would probably be more happy with an Obama appointee than a Romney appointee). But then two things happened.

First, Justice Roberts, a Bush appointee, cast the deciding vote confirming the legality of the health care reform bill’s individual mandate. I have always hoped that once a person makes it to the country’s highest court, to serve pretty much for life, that they leave partisanship at the door. Roberts demonstrated this.

Second, my friend Ron on Google+ pointed out that there will never be a way to break the two party system without some of us “throwing our vote away” on third party candidates. He said he was in it for the long game, and that convinced me. In fact I stated that we should carve a plaque stating that “On this day, someone’s mind was changed via a discussion on the Internet”.

Heh.

And anyway, I’m looking forward to feeling superior no matter who wins. Something has to change, and soon, before we degenerate into throwing shoes.

Let me close with this scary quote from Carl Sagan, taken from his book The Demon-Haunted World and shared with me via G+:

I have a foreboding of an America in my children’s or grandchildren’s time – when the United States is a service and information economy; when nearly all the key manufacturing industries have slipped away to other countries; when awesome technological powers are in the hands of a very few, and no one representing the public interest can even grasp the issues; when the people have lost the ability to set their own agendas or knowledgeably question those in authority; when, clutching our crystals and nervously consulting our horoscopes, our critical faculties in decline, unable to distinguish between what feels good and what’s true, we slide, almost without noticing, back into superstition and darkness.

Mail to hotmail.com accounts being blocked

Just a heads up that I found out today that mail from our mail server is being rejected by hotmail.com mail servers.

One of my mail users was sending a message to a friend and it bounced with:

host mx4.hotmail.com[65.55.92.184] said: 550 SC-001
(SNT0-MC4-F38) Unfortunately, messages from 64.34.199.116 weren’t sent.
Please contact your Internet service provider since part of their network
is on our block list. You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL
FROM command)

So I dutifully contacted our ISP for the mail server, ServerBeach, as well as hotmail. The ServerBeach folks (awesome as always) replied in minutes and said that the IP is not on any other blacklist, so I had to deal directly with hotmail. I got a reply from hotmail that the issue couldn’t be automatically mitigated, so I had to fill out another questionnaire on-line, and I assume I’ll have to wait a couple of days for it to be addressed.

The funny part is that I looked through our logs, and we’ve tried to send exactly five e-mail messages today to hotmail addresses. Two were to the legitimate address that started this whole process, and three were to addresses like:

TanguyKirkpatrick847@hotmail.com
UliYule1780@hotmail.com
FeStoddard952@hotmail.com

which appear to be spammers trying to register on our wiki. The wiki replies and requires an additional action in order to register, and I assume it is this mail traffic that is causing the problem. Note that all of our mailing lists are handled by Sourceforge so this only affects mail from the wiki, project members and employees of OpenNMS.

I think it is pretty ironic that the reason my mail server is being blocked by hotmail is that spammers from hotmail are trying to register on our wiki.

Update: Surprise – I got a rather quick reply from hotmail:

My name is Amrita and I work with the Hotmail Deliverability Support Team.

Your IP (64.34.199.116) was blocked by Hotmail because Hotmail customers have reported email from this IP as unwanted. I have conducted an investigation into the emails originating from your IP space and have implemented mitigation for your deliverability problem. This process may take 24 – 48 hours to replicate completely throughout our system.

Inveneo in American Way Magazine

I’ve been traveling a lot lately, and while I was reading American Way magazine I came across this article “Road Warrior in Search of Internet Connections and Great Beer”.

Now I'm a bit of a road warrior who likes Internet connections and great beer, but alas it wasn't an article about me. It was about Bob Marsh, one of the founders of the non-profit Inveneo which is dedicated to bringing communications technologies to the developing world.

What caught my eye was:

After the 2010 earthquake in Haiti, humanitarian agencies needed instant Internet access where no infrastructure existed. Inveneo had the first Wi-Fi links up in six days.

What wasn’t mentioned is that OpenNMS was being used to monitor those links.

It felt really good to help them out, and we continue to support Inveneo’s efforts still, and it was cool to read more about them.

OpenNMS: MIB Management Through the GUI

Two of the main strengths of OpenNMS as a network management application platform are the fact that it writes to a database and that most of the configuration is done via XML.

The database is sometimes invaluable in addressing a complex problem and while XML isn’t the easiest to use, since it can be parsed by both humans and machines the idea has always been to eventually have a configuration user interface for every file. Thus you could choose which method works best for you.

I like to tell a story of a trip I took to Australia where the client handed me a list of names, usernames, e-mail addresses and phone numbers for over 50 users and asked me to put them in OpenNMS. Rather than struggle to enter them in via the GUI (with some hard to catch transposition errors to be expected) I was able to save the list as a comma separated variable (CSV) file and then use a script to convert that to XML. I just dropped that into the users.xml configuration file and was done.

But some things can be a bit laborious to do by hand and a decent user interface would make things that much easier. One such task is dealing with SNMP MIB files for both events and data collection configuration. So I was very excited when I saw this video that Alejandro put together on his new UI feature.

Check it out. It runs a little over ten minutes and does a great job of demonstrating the functionality. Hats off to Alejandro and the rest of the team for pulling this together. You can find it in the 1.11 releases and it will be included in the next stable version of OpenNMS due in 2013.

OpenNMS: Topology Views

I haven’t been blogging much lately, mainly because I’ve been on the road a lot. It doesn’t look like this is going to change much for the rest of the year, but I am hoping to share more about what is going on in OpenNMS-land.

For the next stable release of OpenNMS, we are aiming to greatly improve aspects of the user interface. We have settled on a technology called Vaadin which should make the UI more like a desktop client than a web page.

A lot of that effort has been focused on a new topology view, i.e a map. I’ve gone on record before as someone who really hates maps. It goes back to some time I spent in 1999 where I was deployed at a client site where I spent hours each day tweaking HP OpenView maps. While the pointy-haired boss types really seem to like maps, they tend to have limited usefulness in actually diagnosing network issues, and quite frequently the effort that goes into maintaining the map isn’t worth it.

So the problem we were trying to solve is: how to make a map that is both useful and maintainable?

Leveraging the fact that OpenNMS is an application platform, Matt wrote a complex piece of code we call the “topology provider”. The idea was that different people need to see the network in different ways, so rather than limit things to one view, why not create a generic topology provider that can be customized to fit those needs. You could look at the network from a Layer 2 and/or Layer 3 perspective (i.e. most maps), or you could choose a geographical view, or a business function view, etc.

The development team in Fulda, Germany, ran with this and decided to create a VMWare VSphere view. With this topology provider, VSphere is queried in order to gather a layout of virtual machines, both host systems and guests, as well as their network attached storage.

One nice feature of the new topology view is what we are calling “semantic zooming”. While you can zoom around the map like you are used to, you can also zoom in and out from a semantic perspective. For example, here is a top level view of three separate VSphere instances.

As you zoom in, you get to see the main host machines as well as other network details:

Zooming in farther brings up a much greater level of detail:

And then you can use traditional zooming to zero in on a machine that is having issues:

The new topology view also provides context sensitive menus. In this case you can bring up the ability to SSH into the system that is having problems:

This is a web-based SSH client that proxies through the OpenNMS system, so you can reach devices on your network even if you are accessing OpenNMS over a VPN or NAT (of course, you can remove this if you think it poses a security risk).

This is available in the current 1.11 development (unstable) branch of OpenNMS, and we are improving it every day. This is just one of the changes we hope will improve the usability of OpenNMS. I’ll post another one tomorrow.

Zipcar

I travel a lot, and one thing I really hate to do is to rent a car. My usual trip is a week or so at a client site deploying OpenNMS, and outside of work and dinner I pretty much just live in the hotel, so there isn’t much driving around that I need to do. However, at certain sites you just can’t avoid driving so I just suck it up and pay something like $5 a mile for the privilege.

Enter Zipcar.

A couple of years ago I was planning on being in Portland, Oregon, probably for OSCON. Portland has great public transportation, so I hate renting a car when I go there, but I wanted to visit a client across the river in Vancouver, Washington. So I checked out the Zipcar service. Zipcar lets you rent cars by the hour. It includes fuel and insurance (with a $750 deductible) and it is supposed to be pretty easy to pick up and drop off. I figured I’d rent a car for a few hours, drive over and drive back, all for less hassle than a usual car rental.

So I signed up. It costs $50 a year for a membership and then rentals are an hourly rate on top of that. For whatever reason, I didn’t use Zipcar on that trip, but I’ve kept my membership current and I travel with my card.

Last week I was in Chicago. I love Chicago, especially since I can get around pretty easily on public transit. They have a great subway system supplemented with lots of bus service. However, I ended up having to go to Loves Park, Illinois, not realizing it is practically in Wisconsin. I pretty much decided that I couldn’t make the trip until I remembered Zipcar.

I was staying on the Magnificent Mile, and the Zipcar website told me that there was a location a couple of blocks away at the Four Seasons hotel. I booked a car for the afternoon, setting up my reservation on-line (I later downloaded the Android app) and when the time came I used the detailed instructions on the website to find the parking garage and my car.

The Zipcar membership card uses RFID, and there is a sensor mounted to the driver’s side windshield of the car. You use your card to both unlock and lock the vehicle, as the key is supposed to stay with the car at all times (mine was attached to the dash using a retractable pull string). Once I got settled in the car, I used the access card in a pocket in the visor (right next to a gas charge card) to exit the garage.

The trip to Loves Park was uneventful, and it would have been impossible to get there on public transit.

Included in the car rental is fuel. The Zipcar philosophy is all about sharing and courtesy, so you are supposed to return the car with at least a quarter of a tank of fuel. I estimated that I would have at least that much left after I returned to Chicago, but I wanted to try out their gas purchasing system, so I filled the tank in Loves Park (I also figured the cost would be less than in Chicago proper).

Each Zipcar comes with a gas card. It works like a credit card, but when you use it the pump asks you for the odometer reading and your membership (or “driver”) number. I’m not sure if this is something unique to Zipcar or if this is a common system usable by any motor fleet, but as soon as I entered the information I was able to fill the tank.

Oh, you might be wondering what type of car Zipcar provided to me. It was a BMW 328i.

I was requested to use premium gasoline.

Some extra nice touches: there was a micro-USB charger in the glovebox, which I ended up using since I needed my phone to act as a GPS (sorry iPhone users – there was no plug for you). There was also a pretty decent mixed CD in the CD player (burned on a CD-ROM). I don’t know if they come with all Zipcars or someone just left them there, but I did the same to keep within the Zipcar spirit.

The only downside was that there was something a little off with the alignment and there was a pronounced vibration while driving. It was very noticeable in the 55-65 mph range, and I dutifully tested a variety of speeds (strictly as a matter of science) but it never fully went away. I can’t seem to find any way to report this to Zipcar, however. While I seem to remember seeing something on my phone during the reservation, I was trying very hard to get the car back in time for the next renter that I didn’t explore it at the time, and now the option is gone. I did manage to get back under the 180 mile/day limit, clocking in at 174.

Outside of the mileage limit (which won’t affect most renters) the other thing to watch out for is that there is no refund if you return the car early, and while I am not sure of the penalties, the website hints that it is very uncool to return a car late, especially if someone has a reservation after you.

Overall the experience was so nice that I’m thinking that NASDAQ:ZIP at $6.50 is probably a deal. It was way more convenient than any car rental I have done in my life, and that is actually saying something.

Oh, are any of my three readers going to the LISA conference in San Diego in December? I won’t be at the conference but I plan to be in the city, and I’ll be using Zipcar to get around if anyone wants to get together.

[UPDATE: I got a surprise call this morning from Nef, the Chicago Zipcar fleet manager. He apologized for the alignment issues I was experiencing and even gave me a credit against future rentals.

What I love about this is that a) someone found this post, b) cared enough to read it and then c) bothered to look up my phone number and call me. I’m definitely looking forward to my next opportunity to use their service]

Ever Wonder What Your Support Dollar Buys?

One of the hardest parts of our business is justifying the purchase of support and services for our open source product. Shouldn’t it “just work”? If I have a bunch of smart people working for me, shouldn’t I be able to figure this out on my own?

The issue is that with a product as powerful and complex as OpenNMS, quite often it is something outside of the application that is causing the problem. For example, I am visiting a large telecom provider this week and we spent part of my time figuring out a complex issue with syslogs. It had nothing to do with OpenNMS and everything to do with their various devices each sending in logs with different formats (sometimes two or more formats from the same type of equipment). I doubt any solution without the flexibility of OpenNMS would have been able to solve it, and the customer told me today “we couldn’t have gotten this to work without you”.

Also, today Jeff was dealing with a support issue with another one of our clients. Their large provisioning import was never completing. He dug around and posted this reply:

Watching the logs with Provisiond turned up to DEBUG, I noticed a single pattern accounting for nearly all the messages in provisiond.log:

2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.1
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.2
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.3
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.4
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.5
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.6
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SnmpWalker: Sending tracker pdu of size 6
2012-10-17 19:20:40,090 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SnmpWalker: Received a tracker PDU of type RESPONSE from /172.22.66.210 of size 0, errorStatus = 0, errorStatusText = Success, errorIndex = 0

It’s the same six objects being requested over and over from ten hosts, all of which appear to be Eltek Valere power units. See there how the size of the RESPONSE PDU is listed as zero? That seemed odd, so I captured some of the SNMP traffic and loaded up the dump into Wireshark.

These devices are replying to our BULK-GETs with response PDUs containing no varbinds, but also indicating no errors, which is silly and seems to send our SnmpWalker class into an infinite loop. You can reproduce this problem using the Net-SNMP snmpbulkget utility:

snmpbulkget –verbose -v2c -c public -Cn6 -Cr1 172.22.66.106 .1.3.6.1.2.1.1.1 .1.3.6.1.2.1.1.2 .1.3.6.1.2.1.1.3 .1.3.6.1.2.1.1.4 .1.3.6.1.2.1.1.5 .1.3.6.1.2.1.1.6

Falling back to SNMPv1 and GET-NEXT seems to elicit a valid response. So I’ve done that for these ten nodes.

This is an example of why we at The OpenNMS Group only hire highly experienced people – how long would it have taken the average person to figure out that out of 7000+ devices these ten were the culprits, as well as coming up with a workaround?

This customer has a bunch of talented people working for them, but they are focused on that client’s business and aren’t as expert on OpenNMS as we are. They might have figured this out, but it could have taken days. Outside of the salaries for that time, the business would suffer since the solution wouldn’t be working. This case on its own probably justified the cost of support.

Now someone might say that a commercial product wouldn’t have suffered from this problem. I find that hard to believe, as any device that abuses the standard to this degree would cause problems for any application. Plus, commercial vendors view support as a cost center, not a revenue stream, and the chance that you would have gotten someone knowledgable on the first try is slim. So you are back to wasting time, and time is money.