Month: May 2017

Server Room Nightmares

Posted on by Tarus

I’m interested in any server room nightmares people would like to share.

Here’s one of mine.

We are in the process of moving offices from Pittsboro, NC down the road to Apex. Unfortunately, we are having some issues getting Spectrum Enterprise to complete the fiber installation at the new place, so while we are out of our old building the lack of network access in the new building means we have a bunch of servers in the old location.

Today while I was working in the new office and mooching of our kind neighbor’s wi-fi, I got several notices that links had failed.

linkDown event list

These were some workstations that we use for training, but when they are not in use we use them as part of our continuous improvement Bamboo farm. I immediately hopped on our Mattermost IT channel and asked if anyone was rebooting or otherwise messing with the machines, and when the answer was “no” I started to investigate.

One suggestion was that the air conditioning may have failed and those machines shut down from overheating. It has happened in the past, but it was both rather cool today and other machines that are more sensitive to such things were still running. I checked it out anyway using our AKCP probe.

temperature graph

The temperature had increased a bit, but it wasn’t anything that should have caused problems (it was caused by the server room door being left open).

Being 30 minutes away, I decided to text my friend Donnie, who is technically gifted as well as working in our old location, and he went to investigate.

For some reason, those three machines had been disconnected from the switch.

Now just for this situation we have an Arlo camera installed in the server room, so using the time stamp on the linkDown traps I found the following video.

Note the slightly balding guy in the red shirt in the lower left corner of the video. He is busy unplugging our devices.

Why? I have no idea. These people represent the IT people for the new tenant, and I assume they had legitimate reasons for being in the server room but messing with our equipment was not one of them.

Seriously, in over 30 years of working with computers, I’ve never heard of anyone going into someone’s house, office, server room or data center and just start unplugging cables. I still have not heard an explanation, but the landlord has had a discussion with the new tenant and it shouldn’t be happening again. It is one reason the important stuff is in that locked half-rack seen in the upper left corner of the video, and the really important stuff is hosted elsewhere.

I am curious – I’m certain this pales compared to other stories out there. Do you have any whoppers to share?

New Meridian® Releases Available

Posted on by Tarus

Just a quick note to point out that new Meridian releases are now available: 2015.1.5 and 2016.1.5

For those who aren’t aware, Meridian is a subscription-based version of OpenNMS built to complement Horizon, the cutting edge release. You can think of it as Meridian is our Red Hat Enterprise Linux to Horizon’s Fedora. There is one major Meridian release per year and each major release is supported for three years.

Before the Meridian/Horizon split it was taking us 18 months or so to do a new major release of OpenNMS. Now we do three to four Horizon major releases a year.

About half of our revenue comes from support contracts and so we had to be extra careful when doing a release, and even with that many of our customers were reluctant to upgrade because the process could be involved. This was bad for two main reasons: often they wouldn’t get bug fixes which meant an increase in support tickets, and more importantly they might miss security updates.

Updates to Meridian, within a major release, are dead simple. This is the process I used yesterday to upgrade our production instance of OpenNMS.

First, I made a backup of the /opt/opennms/etc and /opt/opennms/jetty-webapps/opennms directories. The first is out of habit since configuration files shouldn’t change between point releases, but the second is to preserve any customizations made to the webUI. I modify the main OpenNMS page to include a “weather widget” and that customization gets removed on upgrades. Most users won’t have an issue but just in case I like having a backup.

Next, I stop OpenNMS and run yum install opennms which will download and install the new release. The final step is to run /opt/opennms/bin/install -dis to insure the database is up to date.

And that’s it. In my case, I copy the index.jsp from my backup to restore the weather information, but otherwise you just restart OpenNMS. The process takes minutes and is basically as fast as your Internet connection.

If you have a Meridian subscription, be sure to upgrade as soon as you are able, and if you don’t, what are you waiting for? (grin)

OpenNMS Team Wins 5000€ Prize at TM Forum {open}:hack

Posted on by Tarus

A group of four students from Southampton Solent University, mentored by Dr. Craig Gallen, used OpenNMS to win the top prize at the TeleManagement Forum {open}:hack competition at the TM Forum Live conference in Nice, France.

{open}:hack Winners

Now, a little background is in order. Dr. Gallen founded Entimoss, our OpenNMS partner in the UK and Ireland. He got involved with OpenNMS over a decade ago when he was working on his doctoral thesis entitled “Improving the Practice of Operations Support Systems in the Telecommunications Industry using Open Source”.

Most of his work was focused on a business solution framework called NGOSS (now Frameworx) developed by the TM Forum for creating next generation OSS/BSS software and systems. Now the TM Forum is the world’s leading trade organization for telecommunications providers and at the time was not very friendly toward open source. He demonstrated how an open source platform like OpenNMS could be used to integrate with and tie together these different interfaces to build a reference implementation for part of the framework. Open source was a new concept for the industry, and we were branded the “open source pirates” at first. But Craig persisted, and in 2011 he was awarded the TM Forum’s Outstanding Contributor Award.

In addition to his persistence and ability to deal with large organizations, Craig is also a great teacher. When the TM Forum introduced its {open}:hack program, he wanted to get involved and he found several interested students at Southampton Solent University.

The goals of {open}:hack are:

  1. Accelerate industry deployment of Forum Open APIs, metamodels and architecture across the industry
  2. Validate existing APIs and provide feedback for future iterations to technical collaboration teams
  3. Create IoT/Smart City & NFV/SDN solutions leveraging the Forum Open APIs
  4. Accelerate the incubation of new digital business opportunities in the areas of 5G Network Services & IoT/Smart City
  5. Create extensions to Forum Open APIs to be shared with industry

Participants were given access to APIs from the TM Forum, Huawei, Salesforce and Vodafone, which included things like data from drones, and tasked with creating something beneficial. Their project was called “Port-o-matic” which created an application for accessing services at shipping ports, as well as measuring environmental factors such as pollution. This was especially relevant to them since Southampton is the UK’s number one cruise port and second largest container port (the Titanic set sail from there).

{open}:hack architecture

Their solution leveraged the power of the OpenNMS platform to tie all of these APIs together and then to provide aggregated data to their web application. It can scale to almost any size using the new OpenNMS “Minion” feature which can distribute data collection and monitoring out to the edges of a network, offloading the need to have all of the functionality in a central location and positioning OpenNMS for the Internet of Things (IoT).

The hardest thing to get across to people new to OpenNMS is that it is a platform and not strictly an application. The learning curve can be steep and it is hard to see its value straight out of the box. I love the fact that solutions like the “Port-o-matic” demonstrate the power of OpenNMS.

It is also interesting to note that the second place prize went to a team from Red Hat. For an organization like the TM Forum that was wary of open source to demonstrate such a change of heart is encouraging, and I credit Dr. Gallen with a lot of that advancement.

{open}:hack Group Photo

So congratulations to Joe Appleton, Jergus Lejko, Michael Sievenpiper and Marcin Wisniewski, the winners of this latest {open}:hack competition, and I look forward to seeing more great things from you in the future.

2017 Red Hat Summit

Posted on by Tarus

I had never been to a Red Hat Summit before this year. We are exploring running OpenNMS on OpenShift and so Jesse, David and I decided to head to Boston to see what all the fuss was about.

RHSummit - Airline Sign

I noticed a couple of things are different about visiting Boston in spring versus winter. First of all, the weather was quite nice, and second, Boston can be freakin’ expensive.

And Red Hat spared no expense on this conference. This is the premiere event for companies in the Red Hat ecosystem and they obviously wanted to make an impression. I’m an “old guy” and I can remember going to huge shows put on by HP and IBM and this was on par. It took place at the Boston Convention and Exhibition Center (BCEC) which takes up about a half a million square feet. Red Hat used all of it.

RHSummit - Convention Center Sign

Nothing quite demonstrated the size of this conference than the main auditorium. The centerpiece was a huge screen for the presentation flanked by two smaller screens to show the speaker. That was needed since the place was so big you could barely see the person talking.

RHSummit - Main Auditorium Screen

In addition to the general sessions, there were a large number of talks on pretty much anything related to Red Hat products, philosophy and partners. As a major player in “the cloud” there was a lot of emphasis on OpenShift and OpenStack, but the whole range of offerings was covered from Fedora and CentOS to JBoss and Gluster.

As with most tech conferences, there was an expo floor. This one was dominated by the color red.

RHSummit - Expo Floor

I spent a lot of time wandering around talking with people. Over the years a large number of my friends have been hired by Red Hat, and as I’ve curtailed my participation in a lot of the big Linux conferences, it was nice to see them again. I ran into Brian Proffitt and Ruth Suehle near the center of the expo:

RHSummit - Brian Proffitt and Ruth Suehle

It was also nice to run into the Latvian army. The Zabbix crew had a booth and it was cool to see Alexei and Alex again, although it was ironic that I missed them on my trip to Riga (they were actually driving north to Tallinn when I was heading south).

RHSummit - Zabbix Booth

Zabbix, like OpenNMS, is 100% open source and thus not only do we get along, I quite like them and look forward to chatting about the joys and challenges about running an open source business when we meet.

Speaking of meeting, I also got to meet Brian Stinson of the CentOS project.

RHSummit - Brian Stinson from CentOS

We swapped some stories and recounted the strange and funny time when Jerry Taylor, the City Manager of Tuttle, Oklahoma, claimed the CentOS project had hacked his city’s website. Has it been eleven years? Wow.

As part of the conference, Red Hat provided lunch. It was always a pretty hectic time since the show was packed and nothing demonstrated this more than trying to serve lunch to all those people.

RHSummit - Lunch Crowd

As far as conference lunches go, it was above average, but I did find it funny that they only served water to drink (usually there are cans of soda, etc.) I overhead one Red Hat employee say to another, you know, we can afford that gigantic screen but all we get is water?

On Wednesday night, Red Hat purchased a ton of tickets to the Red Sox game at Fenway Park. While I can’t find a reference to actual conference attendance figures, I heard the number 5000 being batted around which was a significant portion of the ballpark (it holds a little over 37,000). They gave us all red baseball caps and you could definitely see them in the crowd.

RHSummit - Fenway Park

For our annual developers conference, Dev-Jam, we have about one-one hundredth the amount of people to see the Twins play, but we also get better seats. (grin)

It was my first time at the historic Fenway Park, and the fans were almost more fun to watch than the game. I also enjoy trying to explain the game of baseball to people from outside the country, and this was made more interesting by some bad blood between the Sox and the Orioles that resulted in the ejection of the Orioles’ pitcher for hitting a batter.

Fenway is relatively close to Cambridge, so I took the opportunity to visit a friend of mine who is a professor. I decided to walk to Harvard Square along the river, where the rowing teams were practicing.

RHSummit - Rowing

Now whenever I see a movie featuring Ivy League students on the water, I’ll know where that was shot.

It was also nice to be able to spend some time with David and Jesse. While I work with David almost daily, we’re so busy that it is hard to find time to talk strategy and plan for the future of OpenNMS. Jesse, our CTO, moved back to Canada after the birth of his son to be closer to family, and it was also nice to have time to spend with him. Walking to dinner one night David took this picture

RHSummit - River and Bridge

which turned out so much better on his iPhone 6S than my Nexus 6P.

I often say that Red Hat, as a company, doesn’t get the credit it deserves since it is headquartered in North Carolina and not Silicon Valley. Our companies share a similar philosophy of taking care of customers, creating great open source software and producing steady growth, versus, say, chasing unicorns. It was wonderful to see that work demonstrated in such a large and professional conference, and I hope next year I’ll get to speak (although I doubt it will be on the big stage).

Fifteen Years

Posted on by Tarus

On Sunday my mother celebrated her 75th birthday.

Although a happy occasion, why is this relevant to an open source blog? Well, it was soon after her 60th birthday in 2002 that I started my first company around OpenNMS.

I did not start OpenNMS, it began in the summer of 1999, with the first code posted on Sourceforge in March of 2000 by a company called Oculan. I started working with Oculan in September of 2001, and in May of 2002 they decided to stop contributing to OpenNMS. I saw the potential, so I asked Steve Giles, the founder and CEO, if I could have the OpenNMS project. He looked at his watch and said if I was off his payroll by Friday, he’d give me the domain names, a couple of servers, and he would sprinkle water on me and I would be the new OpenNMS maintainer.

That was actually the easy part. Explaining to my wife that I had quit my job and started a company “selling free software” was a bit harder.

sortova.com from archive.org circa May 2002

And thus Sortova Consulting Group was born. It was named after my farm. When Andrea and I decided we wanted to have a farm, we first bought raw land. In driving out from Raleigh to work on it we would pass this little farm with a barn, some cows, etc., and on the mailbox was a sign reading “Almosta Farm”. I joked that if that was “almost a farm” then what we had was just “sort of a farm”. Later, when we bought the place where we still live, the name Sortova Farm stuck.

We pronounce it “Sore-toe-va”. Only one customer ever pulled me aside and asked if it really meant “sort of a” consulting group. He laughed when I confirmed that it did.

Considering that I didn’t have any prior business experience, Java experience, or even real Internet access at my home, it is amazing that OpenNMS survived to this day. It is a wonder what you can accomplish with pure stubbornness.

Now my one true superpower is my ability to get the most fantastic people on the planet to work with me. The first group of those came from the OpenNMS community. When I was running Sortova it was the gang that later became the Order of the Green Polo that kept me going, mainly through mailing lists and IRC. In September of 2004 my good friend and business partner David Hustace and I founded the OpenNMS Group, and that corporation is still going strong. In 2009 we mortgaged our houses to buy the copyright to the Oculan OpenNMS code and thus brought all of it back under one organization, and two of the original OpenNMS team at Oculan now work for OpenNMS.

When I visit Silicon Valley I often get to meet some brilliant people, but the joy of this can be offset by the pervasive attitude of focusing on technology simply to make money. I know of a number of personally successful people who built companies, sold them, and then those products vanished into obscurity. Remember VA Linux? Their stock rose over 700% on the first day of trading, but where are they now? Did they ever deliver on their promises to the stockholders?

I want to build with OpenNMS something that will last well beyond my involvement with the project. I’ve gotten it to the point where I am not longer expressly required to make it thrive, but I am still working on its legacy. We want it to be nothing less than the de facto standard for monitoring everything, which is a high bar.

Note that I still would like to make a lot of money, but that isn’t the core driving force of the business. Our mission statement is “Help Customers – Have Fun – Make Money” in that order. If you have happy customers and happy employees, the money will come.

Fifteen years ago I made a leap of faith, in both myself, my family and my friends. I’m extremely happy I did.

Privacy and Trash

Posted on by Tarus

Meet Sam. Sam is in his early twenties and grew up in Lake Mills, Wisconsin. He graduated from the University of Wisconsin in Madison in 2012. He is currently on vacation in Athens, Greece, with his girlfriend Sara. They managed to find an amazing deal on American Airlines from Minneapolis to Athens for $200 for the both of them, but with taxes and fees that ballooned up to nearly $850.

I have a copy of Sam’s resume, his Gmail address and his phone number. I know how long he’ll be gone and what seats they will be sitting in on their return. In fact, I know a lot more about Sam and Sara (Facebook and its ilk are ubiquitous) but I’m a little uncomfortable revealing as much as I have, so I’ll stop.

It is all because of this:

Sam Boarding Pass

With all the focus recently on the security of devices like those that make up the Internet of Things, what is often forgotten is that traditional paper has huge security issues in today’s connected world.

Airlines still insist on printing first and last names along with record locater codes on boarding passes. That is often all that is required to access a particular reservation. From there you can get information such as e-mail addresses and phone numbers.

This reminds me of when credit cards first came out and to use one the merchant would take an actual imprint of the card on carbon copy paper. Since that included the shopper’s name, complete card number and expiration date, it became easy for thieves to steal this information. At least now almost all receipts include, at most, the last for digits of the card (in case you were wondering, Sam used a Mastercard ending in 3286).

The genesis of this post arose from a more malicious reason. I fly a lot and over the years commercial air travel (which is the only air travel I can afford) has become less of a special occasion and more like taking a bad bus trip. People use the “seat back pocket” as their personal trash can, to the point that I almost never use it myself, even when I get upgraded to first class. Nasty. On this trip, the duration from when the last person got off the inbound plane until we started boarding our flight was less than ten minutes, so trust me when I say little was cleaned between flights.

I don’t blame the airlines. Consumers have spoken, and what they want is cheap airfare, so it is up to us to be respectful of our fellow passengers.

Anyway, when I see folks like Sam leave information like this as trash, I am so tempted to do things like reassign his seat to one in the middle next to the lavatory (it’s an 11 hour flight), or to cancel his flight completely. Lucky for him I believe in karma, and I just can’t bring myself to do it.

The basics of security involve two things: something you have and something you know. We need to apply this to everything that needs to be secure. I get so frustrated with systems in the United States, such as the new “chip” cards being used for credit and debit. Introduced a decade ago in Europe, their systems use “chip and PIN” – something you have, your card, and something you know, your PIN. In the US we are moving to “chip and signature” – something you have, your card, and something anyone can fake in a heartbeat, your signature.

(sigh)

This is especially touchy since two summers ago my spouse had her purse stolen. We immediately canceled and closed all of the accounts, but they were still able to get over $2000 out of our checking account. They used a paper check from another theft and then they cashed it at the bank using her ID. The bank forgot the “something you know” part of security even though they were quite aware that our account had been compromised and the account number changed. Only after the fact did they offer to “flag” transactions on our account for extra scrutiny, and now neither of us carries paper checks, although thieves could probably guess our bank from our ATM debit cards (we did get our money back from the bank).

So be careful. Buy a good shredder. If you need to dispose of paper when traveling, tear it into tiny bits and drop it in the nastiest trash can you can find … and not in the seat back pocket.

LinkedIn

Posted on by Tarus

I’m at Red Hat Summit in Boston this week so expect a longer post on the conference later, but I wanted to mention that I’ve reopened a LinkedIn account after an absence of several years. You can find me here:

https://www.linkedin.com/in/tarusbalog

I left the network due to how they were handling privacy issues. I’m still not 100% happy with it now, but I think I can control how much information I share and I do have a need that I think the service can provide.

I was walking in Boston yesterday and I saw a sign for Harvard Medical School. They used to use OpenNMS and I really enjoyed working with the guys who worked there. Most of them have moved on, so I was curious to know where they were and if they were still in the city. It dawned on me that LinkedIn would have helped in this situation.

I don’t like a number of changes that have been made to the site, such as the inability to feature external links (such as to this blog which will remain one of my main ways to communicate) but it may be just my inability to navigate the website. OpenNMS is also on LinkedIn, and it looks like you can “follow” the company as well:

https://www.linkedin.com/company/the-opennms-group

Anyway, let’s give this a go. See you in the toobz.