OpenNMS Group

It’s Friday, So This Must Be Finland

Posted on by tarus

Chris Dibona once wrote that, on a trip to China, he was “visiting the birthplace of all of my electronics”. On my first trip to Finland I feel a little like I am visiting the birthplace of OpenNMS.

OpenNMS was actually born in North Carolina, USA, but the thoughts that made it possible were the influence of a local company called Red Hat, which in turn was inspired by the work of Linus Torvalds, a Finn who was born in Helsinki.

On a modern note, the new OpenNMS GUI relies heavily on Vaadin, a software framework that was invented in Finland. Finally, my family originated in Hungary. The Hungarian language (which I do not speak) is related to few other languages, but one of those happens to be Finnish.

I came here from Sweden to visit a new client, and that makes Finland the 26th country in which The OpenNMS Group has customers (the others being, in no particular order, Ireland, Spain, Portugal, Egypt, Canada, Mexico, Singapore, Japan, Australia, Israel, Denmark, France, Germany, Switzerland, The Netherlands, the UK, Italy, Trinidad, Malta, India, Honduras, Chile, Sweden, the UAE and the US.)

I got a really good vibe from Helsinki, even though it was really cold. The Germans have a term for temperatures at or below -15C, arschkalt, and I did experience that here.

The client was pretty cool. I met Jari last week at the OpenNMS Users Conference and he and his coworkers went out of their way to may sure I enjoyed my short stay. On Thursday night we went to a restaurant called Salutorget and had a wonderful meal, and on Friday we visited a place called Stone’s Beer and Burger where I had a Finnish “veggie burger” – instead of the usual attempt to make a burger patty out of vegetables, it was a bun and grilled vegetables, which was quite tasty.

The beer had to wait until after work, when we visited Teerenpeli. This was on the advice of Ville, the Vaadin employee who came to the Users Conference and suggested some places I should visit while in Helsinki.

On a somber note, there was a marble memorial to people killed in the Finnish Civil War in the office building where we were working. I thought it unusual that there was a lack of Finnish surnames starting with “B” through “F” – there were three names that started with “A” and then it jumps to “G”.

Overall, the only criticism I have of the trip is that it was too short. Next time I come I hope to take a side trip to Tallin, Estonia and/or St. Petersburg, Russia. Both are surprisingly close, and I need a few more countries to visit. I’m at 33 at the moment and I want to hit 50 before I am 50 years old.

But I think I want the next one to be a little warmer.

Note: I didn’t think of Monty Python’s song “Finland” while I was there. Not once. I swear. Really.

Welcome Eric Evans to the OpenNMS Group

Posted on by tarus

It is with much happiness and joy that I am able to announce that today Eric Evans joins us at the OpenNMS Group.

I’ve known Eric for nearly 11 years, and he was instrumental in bringing OpenNMS to Rackspace (one of our first customers). While I don’t know if OpenNMS would be here if it wasn’t for Rackspace (and thus Eric), I do know that their support of the project was very important in our surviving through those early years.

Eric at an Early Dev-Jam. Photo credit Mike Huot

Since Eric was instrumental in the development of the Rackspace Cloud as well as the person who popularized the term “noSQL” it is fitting that we have brought him on board to architect both our cloud strategy (yes, “cloud”, I said it) as well as making OpenNMS fully distributable so it can scale even more.

Look forward to many great things as we add Eric to a team already full of awesomeness.

Intuit Payroll Hell

Posted on by tarus

Okay, so I was all excited that we were opening an office in Georgia. Over the years we have had employees in other states outside of North Carolina (Ohio, Massachusetts and Georgia) but since we never had an actual office presence in those states, they officially worked out of the NC office, and so I withheld NC taxes and unemployment for them and they filed NC tax returns as out of state workers.

Now that we have an actual office in Georgia, I thought I’d better start withholding GA taxes and unemployment. So I contacted my payroll provider to make this happen.

I have been using Quickbooks on the Mac for bookkeeping for many years now, and their preferred payroll provider was a company called Paycycle. I liked Paycycle and never had much trouble with them, but several years ago they were purchased by Intuit and things have gone downhill.

When I contacted Intuit about adding the new state (which used to be pretty easy under Paycycle) I was told I would need to upgrade my service, basically doubling my cost. Considering the time I would have to spend in switching to another provider, I agreed to the change.

Thus began my descent into Payroll Hell.

The moment they switched my service, my Payroll To Do list filled up with seven years worth of tasks requesting me to enter in state tax information, not only for NC but for OH, MA and GA. As I was heading out at the time for a three week trip to Europe, I didn’t think much of it and figured I could fix it with a phone call when I got back.

I should have found the time.

The night that payroll was due for January, I was in London. Usually the payroll process takes less than five minutes, but this time it wouldn’t complete, with the system throwing a very unhelpful error message. I was lucky that the hotel had decent Internet access, so I was able to contact Intuit support on-line and after about two hours they managed to get the system to allow me to run payroll.

That is when I found out that even though they doubled my fees, they were not going to electronically file my state taxes (sigh).

When I returned to the US I made getting the payroll system fixed a priority. So I called Intuit support again and thought that my request, to simply set the state taxes history to start on 1 January, was a simple one.

I was wrong.

After spending another two hours on the phone, I was told that the only way I could get the system back to normal would be to enter in values for state taxes going back to 2006, for all four states.

Not happening. I told they guy to just put the system back to where it was before, and he said he couldn’t do that as the service would now be “Basic” and not include electronic filing of Federal taxes. Again, all of these different levels of service obviously point to the Intuit influence, as Paycycle was pretty much a devotee of the Apple method of simplicity.

I told the guy to forget it, and I would find another solution.

My first stop was the Bank of America site. I’ve been a client of theirs for decades, but it turns out that their payroll solution is provided by Intuit. No way I was going down that path again, but I did notice that it was considerably cheaper to get it through BofA and that electronic filing of state taxes was included. Another way that long term clients like myself were getting screwed.

When I look for vendors, I like to visit the list of OpenNMS Group customers. It turns out that ADP is a client. Now it is ADP Dealer Services and not the ADP division that would provide my payroll, but I like patronizing companies that patronize us, so I signed up. Our February payroll should be handled by them.

Today I called Intuit to cancel my service. The representative pointed me to a section on the web page where, after logging in, I could cancel the service. So I dutifully filled out the little exit interview, clicked submit and …

It generated an error asking me to fill out the interview I just filled out.

(sigh)

Another call to Intuit, another 30 minutes or so on hold, and I’m told that I have to clear the state tax “to do” items, the very reason I’m canceling the service, before I could cancel the service.

I’m afraid I went a little non-linear.

After my rant, I sat on hold for ten more minutes until I could speak with a supervisor. She was able to cancel my account. My goodness, maybe they should put her on the front lines.

Anyway, just wanted to post my experience in case any of my three readers was considering using Intuit Payroll services. In a word, don’t.

Expansion

Posted on by tarus

Things are really rolling at OpenNMS, and we’re only a few weeks into the year.

I am happy to announce that we have officially opened a regional office in Marietta, Georgia.

The hope is to leverage the talent in the area for some of the positions we have available and to provide an alternative for people who don’t want to relocate to North Carolina.

Also, we are renting out space in that office for co-working, so if you are in the area and would like a cool environment in which to work, drop me a note.

London Training

Posted on by tarus

Last week I had the pleasure of teaching an OpenNMS course in London.

We’ve been doing some work with BT (British Telecom) and part of that involved training. The hardest part in doing training outside of our main office is finding a place that also has computers we can use. It dawned on me that Red Hat has a London office, so I sent a few e-mails and they were gracious enough to loan us both a conference room and a mobile classroom equipped with nice laptops.

I think I need to make it a tradition to visit as many Red Hat offices as I can (grin).

This one was in a building that used to house Special Operations:

The office is on Baker Street (yes, that Baker Street) which is in Mayfair, a very posh section of London. It’s home to Selfridges on Oxford Street, a Harrods-like huge department store, as well as a slew of expensive cars: Bentleys, Rolls, Aston Martins as well as Ferrari’s and Mercedes.

Not that I had too much time to spend looking at the sights. I was there to teach a class and I am in the middle of totally redesigning it, so my first couple of nights were spent working on slides. On Thursday, however, my friend Jonathan Sartin (OGP) had invited me to see a presentation on “The Universe Within” at the Royal Institute by professor Neil Shubin.

We got there a bit early, so I grabbed a local ale and we waited for Craig Gallen (OGP) to show up from the Royal Society (a different “Royal”). The auditorium was pretty full and steeply pitched, which reminded me of some of the many colleges I attended.

Shubin is best known for finding a transitional fossil named Tiktaalik in the arctic region of Canada. A good portion of the hour long presentation was on finding it, starting with looking for fossils along the highways of Pennsylvania. As my family is originally from there, I can remember driving by these hills that were steeply cut so the road would run straight and you could see the different strata plainly. He was looking for fossils in the Devonian era (about 375 million years ago) specifically for an animal that closely resembled a lobed fish but with the flat head and neck of the later land dwelling amphibians. After several years he found it in Tiktaalik.

The talk was good, but at one point he mentioned that the Earth was actually speeding up and the days were getting shorter. This didn’t sound right to me, as I remember once in high school actually calculating the effect the Moon has on the Earth’s rotation and that it was slowing down (through its influence on tides). Furthermore, it would slow down until the rotation of the Earth (a “day”) matched the rotation of the Moon, which is a little more than 27 current days. When that happens, the Moon will spiral into the Earth, causing a few traffic jams and flight delays.

Actually, the calculations showed this was to happen in about 7 billion years, and by that time the Sun will be a Red Giant with a diameter out to about the orbit of Mars, so no worries.

Anyway, when question and answer session happened, I decided to call him on it. He pretty much corrected himself, “It *is* slowing down, the days are getting shorter … I mean longer. Did I say shorter? I meant longer” and left it at that, but Jonathan gave me points for asking a question at the Royal Institute on my first visit, and one meant to correct the speaker at that.

On the way out, I noticed that there were chargers out front for electric cars. Cool.

When I mentioned I was in London, Alex Hoogerhuis, an OpenNMS guru and current resident of Norway, decided to fly in for the day and visit with me. It was fun. We walked down the road to a pub called “The Pheasant” and had a nice meal, and afterward went to the lounge and talked about planes. He taught me the easiest ways to identify the most common commercial jets and I plan to write that up at some point.

The next morning we took a cab to Terminal 5 at Heathrow. Since he no longer has elite status I was able to get him into the BA lounge, and he was able to show me a really cool, almost hidden section of the lounge where we waited for our gates to be announced. It was a fun visit and I hope to be able to see him again soon, which will probably be at the OpenNMS Users Conference in March – the OpenNMS event of the year.

Be there. (grin)

OpenNMS is Hiring

Posted on by tarus

OpenNMS is Hiring!

I hope everyone had a great holiday. I enjoyed it, but I did miss getting a couple of gifts: two top-notch Java developers.

OpenNMS is hiring. In our quest to become the de facto network management platform of choice, we’ve experienced an increased demand for new features and we are now in a position to hire a couple of new developers. We are are looking for both a senior and a more junior person to join our close-knit team to make OpenNMS the best it can be.

If you are interested, you should have wicked Java skills and be able to demonstrate how you’ve used them over the years. OpenNMS heavily leverages the Spring Framework, as well as Hibernate, and we use Maven to build it all. Other technologies we use are (but not limited to) OSGi, Google Web Toolkit (GWT), Jetty, JBoss, Vaadin and Karaf. Yes, we like to play will all the new Java toys.

We also rely on junit tests, so knowledge of Agile methods such as test driven development are a big plus. We are a network management solutions company, so experience with similar products and things like SNMP is valuable. Open source experience will earn you even more points.

For the senior position, you should have that and more (grin).

As one would expect with a job like this, we are looking for people used to being self-directed – we have a pretty flat management structure.

Other things we have: a awesome team. You will be able to work in either the Pittsboro, North Carolina office or the Marietta, Georgia office with some of the best people in the business. OpenNMS has no outside investment and we have an 8 year track record of 25%-40% year over year revenue growth. Our business plan is “Spend Less Than You Earn”. This allows us to be focused on our customers, which are spread out over 25 countries, and our mission statement is “Help Customers, Have Fun, Make Money”.

Since we want our mission to be successful, we do try to take your mind off the small stuff with full health benefits, profit sharing, a 401K plan, and great holiday and vacation policies. We also don’t scrimp on computer equipment. While there are those of us who travel a lot, these positions will require minimal travel.

To help us in our quest to find the brightest and the best, we have retained the services FORTUNE Personnel Consultants. If you are interested, please contact David Cole at david.cole@fpcraleigh.com or +1 (919) 760 3016.

You can also check out http://issues.opennms.com and fix a bug or two. That’s always a good way to get our attention.

Ever Wonder What Your Support Dollar Buys?

Posted on by tarus

One of the hardest parts of our business is justifying the purchase of support and services for our open source product. Shouldn’t it “just work”? If I have a bunch of smart people working for me, shouldn’t I be able to figure this out on my own?

The issue is that with a product as powerful and complex as OpenNMS, quite often it is something outside of the application that is causing the problem. For example, I am visiting a large telecom provider this week and we spent part of my time figuring out a complex issue with syslogs. It had nothing to do with OpenNMS and everything to do with their various devices each sending in logs with different formats (sometimes two or more formats from the same type of equipment). I doubt any solution without the flexibility of OpenNMS would have been able to solve it, and the customer told me today “we couldn’t have gotten this to work without you”.

Also, today Jeff was dealing with a support issue with another one of our clients. Their large provisioning import was never completing. He dug around and posted this reply:

Watching the logs with Provisiond turned up to DEBUG, I noticed a single pattern accounting for nearly all the messages in provisiond.log:

2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.1
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.2
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.3
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.4
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.5
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SingleInstanceTracker: Requesting oid following: .1.3.6.1.2.1.1.6
2012-10-17 19:20:40,087 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SnmpWalker: Sending tracker pdu of size 6
2012-10-17 19:20:40,090 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0]SnmpWalker: Received a tracker PDU of type RESPONSE from /172.22.66.210 of size 0, errorStatus = 0, errorStatusText = Success, errorIndex = 0

It’s the same six objects being requested over and over from ten hosts, all of which appear to be Eltek Valere power units. See there how the size of the RESPONSE PDU is listed as zero? That seemed odd, so I captured some of the SNMP traffic and loaded up the dump into Wireshark.

These devices are replying to our BULK-GETs with response PDUs containing no varbinds, but also indicating no errors, which is silly and seems to send our SnmpWalker class into an infinite loop. You can reproduce this problem using the Net-SNMP snmpbulkget utility:

snmpbulkget –verbose -v2c -c public -Cn6 -Cr1 172.22.66.106 .1.3.6.1.2.1.1.1 .1.3.6.1.2.1.1.2 .1.3.6.1.2.1.1.3 .1.3.6.1.2.1.1.4 .1.3.6.1.2.1.1.5 .1.3.6.1.2.1.1.6

Falling back to SNMPv1 and GET-NEXT seems to elicit a valid response. So I’ve done that for these ten nodes.

This is an example of why we at The OpenNMS Group only hire highly experienced people – how long would it have taken the average person to figure out that out of 7000+ devices these ten were the culprits, as well as coming up with a workaround?

This customer has a bunch of talented people working for them, but they are focused on that client’s business and aren’t as expert on OpenNMS as we are. They might have figured this out, but it could have taken days. Outside of the salaries for that time, the business would suffer since the solution wouldn’t be working. This case on its own probably justified the cost of support.

Now someone might say that a commercial product wouldn’t have suffered from this problem. I find that hard to believe, as any device that abuses the standard to this degree would cause problems for any application. Plus, commercial vendors view support as a cost center, not a revenue stream, and the chance that you would have gotten someone knowledgable on the first try is slim. So you are back to wasting time, and time is money.

Happy Birthday to Us

Posted on by tarus

Just a quick note that Saturday marked the 8th anniversary of The OpenNMS Group.

David Hustace, Matt Brozowski and myself formed the company with a start date of 1 September 2004. It’s been an amazing ride and we’re just getting started.

When Vendors Become Customers

Posted on by tarus

I just wanted to post a little note on something I find encouraging.

As I was doing some billing today, Quickbooks complained, twice, that the customer I was billing was already in the system – as a vendor.

It’s cool when you get to do business with your clients, and vice versa.

Ubuntu FTW

Posted on by tarus

Okay, I’m both tired and hoarse. I’m hoarse from teaching class all day, and I’m tired because I didn’t get to bed until 1am last night.

The last part was due to my inability to get Fedora to run on our classroom machines.

Usually, it is quite simple. I do a fresh install of Fedora, yum upgrade, add the repos for rpmfusion, install the drivers for the ATI cards and the Broadcom wireless interface and I’m good to go.

Starting at about 11am on Sunday, I downloaded Fedora 16 and installed it. This time it complained about the need of a 1-2 MB “BIOS” partition – something about how Fedora is handling GPT partitioned disks. Anyway, that didn’t seem to work since every time I tried to boot it ended up booting from the wrong partition (into Windows).

I decided to punt on Fedora 16 and went back to Fedora 15. That seemed to work okay, but after the couple of hours it took to install and upgrade, it turns out that the rpmfusion packages have, once again, not been updated to match the current kernel. It also seems impossible to set it up so that the correct kernel can be installed. As I mentioned above, usually the kmod packages “just work” so I was stuck trying the akmod packages which, in my experience, never work.

By now it’s time for supper, and when we got back I decided to try Ubuntu (I had five students showing up Monday morning and I needed something). I used to use Ubuntu years ago as the training distro, but Canonical made a questionable hiring decision and I ripped it out. That person flamed out pretty spectacularly, and since I’ve been using Ubuntu over the last nine months or so off and on I was curious as to how easy it would be to install.

Piece of pie. Easy as cake.

With the exception of having to add the “nomodeset” option to the initial kernel boot, it “just worked”, but it took us several hours to finish all six machines.

(sigh)

So, it’s another win for Ubuntu – my new default distro for training.

As far as the class goes, we have a great group. We have two people who are somewhat local coming up from Lumberton, NC. We also have one guy from Chicago who works for Sears (a commercial support customer) and two people all the way from Mellerud, Sweden – one of whom was a FIFA referee for five years. I keep threatening to hand out red cards.

It should be a fun week. Jeff is team teaching this with me, so I believe my voice will hold out.

I’m not sure if the class is rooting for that, however.