Thoughts on Security and Open Source Software

Due to the recent supply-chain attack on Solarwinds products, I wanted to put down a few thoughts on the role of open source software and security. It is kind of a rambling post and I’ll probably lose all three of my readers by the end, but I found it interesting to think about how we got here in the first place.

I got my first computer, a TRS-80, as a Christmas present in 1978 from my parents.

Tarus and his TRS-80

As far as I know, these are the only known pictures of it, lifted from my high school yearbook.

Now, I know what you are thinking: Dude, looking that good how did you find the time off your social calendar to play with computers? Listen, if you love something, you make the time.

(grin)

Unlike today, I pretty much knew about all of the software that ran on that system. This was before “open source” (and before a lot of things) but since the most common programming language was BASIC, the main way to get software was to type in the program listing from a magazine or book. Thus it was “source available” at least, and that’s how I learned to type as well as being introduced to the “syntax error”. That cassette deck in the picture was the original way to store and retrieve programs, but if you were willing to spend about the same amount as the computer cost you could buy an external floppy drive. The very first program I bought on a floppy was from this little company called Microsoft, and it was their version of the Colossal Cave Adventure. Being Microsoft it came on a specially formatted floppy that tried to prevent access to the code or the ability to copy it.

And that was pretty much the way of the future, with huge fortunes being built on proprietary software. But still, for the most part you were aware of what was running on your particular system. You could trust the software that ran on your system as much as your could trust the company providing it.

Then along comes the Internet, the World Wide Web and browsers. At first, browsers didn’t do much dynamically. They would reach out and return static content, but then people started to want more from their browsing experience and along came Java applets, Flash and JavaScript. Now when you visit a website it can be hard to tell if you are getting tonight’s television listings or unknowingly mining Bitcoin. You are no longer in charge of the software that you run on your computer, and that can make it hard to make judgements about security.

I run a number of browsers on my computer but my default is Firefox. Firefox has a cool plugin called NoScript (and there are probably similar solutions for other browsers). NoScript is an extension that lets the user choose what JavaScript code is executed by the browser when visiting a page. A word of warning: the moment you install NoScript, you will break the Internet until you allow at least some JavaScript to run. It is rare to visit a site without JavaScript, and with NoScript I can audit what gets executed. I especially like this for visiting sensitive sites like banks or my health insurance provider.

Speaking of which, I just filed a grievance with Anthem. We recently switched health insurance companies and I noticed that when I go to the login page they are sending information to companies like Google, Microsoft (bing.com) and Facebook. Why?

Blocked JavaScript on the Anthem Website

I pretty much know the reason. Anthem didn’t build their own website, they probably hired a marketing company to do it, or at least part of it, and that’s just the way things are done, now. You send information to those sites in order to get analytics on who is visiting your site, and while I’m fine with it when I’m thinking about buying a car, I am not okay with it coming from my insurance company or my bank. There are certain laws governing such privacy, with more coming every day, and there are consequences for violating it. They are supposed to get back to me in 30 days to let me know what they are sending, and if it is personal information, even if it is just an IP Address, it could be a violation.

I bring this up in part to complain but mainly to illustrate how hard it is to be “secure” with modern software. You would think you could trust a well known insurance company to know better, but it looks like you can’t.

Which brings us back to Solarwinds.

Full disclosure: I am heavily involved in the open source network monitoring platform OpenNMS. While we don’t compete head to head with Solarwinds products (our platform is designed for people with at least a moderate amount of skill with using enterprise software while Solarwinds is more “pointy-clicky”) we have had a number of former Solarwinds users switch to our solution so we can be considered competitors in that fashion. I don’t believe we have ever lost a deal to Solarwinds, at least one in which our sales team was involved.

Now, I wouldn’t wish what happened to Solarwinds on my worst enemy, especially since the exploit impacted a large number of US Government sites and that does affect me personally. But I have to point out the irony of a company known for criticizing open source software, specifically on security, to let this happen to their product. Take this post from on of their forums. While I wasn’t able to find out if the author worked at Solarwinds or not, they compare open source to “eating from a dirty fork”.

Seriously.

But is open source really more secure? Yes, but in order to explain that I have to talk about types of security issues.

Security issues can be divided into “unintentional”, i.e. bugs, and “intentional”, someone actively trying to manipulate the software. While all software but the most simple suffers from bugs, what happened to the Solarwinds supply chain was definitely intentional.

When it comes to unintentional security issues, the main argument against open source is that since the code is available to anyone, a bad actor could exploit a security weakness and no one would know. They don’t have to tell anyone about it. There is some validity to the argument but in my experience security issues in open source code tend to be found by conscientious people who duly report them. Even with OpenNMS we have had our share of issues, and I’d like to talk about two of them.

The first comes from back in 2015, and it involved a Java serialization bug in the Apache commons library. The affected library was in use by a large number of applications, but it turns out OpenNMS was used as a reference to demonstrate the exploit. While there was nothing funny about a remote code execution vulnerability, I did find it amusing that they discovered it with OpenNMS running on Windows. Yes, you can get OpenNMS to run on Windows, but it is definitely not easy so I have to admire them for getting it to work.

I really didn’t admire them for releasing the issue without contacting us first. Sending an email to “security” at “opennms.org” gets seen by a lot of people and we take security extremely seriously. We immediately issued a work around (which was to make sure the firewall blocked the port that allowed the exploit) and implemented the upgraded library when it became available. One reason we didn’t see it previously is that most OpenNMS users tend to run it on Linux and it is just a good security practice to block all but needed ports via the firewall.

The second one is more recent. A researcher found a JEXL vulnerability in Newts, which is a time series database project we maintain. They reached out to us first, and not only did we realize that the issue was present in Newts, it was also present in OpenNMS. The development team rapidly released a fix and we did a full disclosure, giving due credit to the reporter.

In my experience that is the more common case within open source. Someone finds the issue, either through experimentation or by examining the code, they communicate it to the maintainers and it gets fixed. The issue is then communicated to the community at large. I believe that is the main reason open source is more secure than closed source.

With respect to proprietary software, it doesn’t appear that having the code hidden really helps. I was unable to find a comprehensive list of zero-day Windows exploits but there seem to be a lot of them. I don’t mean to imply that Windows is exceptionally buggy but it is a common and huge application and that complexity lends itself to bugs. Also, I’m not sure if the code is truly hidden. I’m certain that someone, somewhere, outside of Microsoft has a copy of at least some of the code. Since that code isn’t freely available, they probably have it for less than noble reasons, and one can not expect any security issues they find to be reported in order to be fixed.

There seems to be this misunderstanding that proprietary code must somehow be “better” than open source code. Trust me, in my day I’ve seen some seriously crappy code sold at high prices under the banner of proprietary enterprise software. I knew of one company that wrote up a bunch of fancy bash scripts (not that there is anything wrong with fancy bash scripts) and then distributed them encrypted. The product shipped with a compiled program that would spawn a shell, decrypt the script, execute it and then kill the shell.

Also, at OpenNMS we rely heavily on unit tests. When a feature is developed the person writing the code also creates code to “test” the feature to make sure it works. When we compile OpenNMS the tests are run to make sure the changes being made didn’t break anything that used to work. Currently we have over 8000 of these tests. I was talking to a person about this who worked for a proprietary software company and he said, “oh, we tried that, but it was too hard.”

Finally, I want to get back to that other type of security issue, the “intentional” one. To my understanding, someone was able to get access to the servers that built and distributed Solarwinds products, and they added in malware that let them compromise target networks when they upgraded their applications. Any way you look at it, it was just sloppy security, but I think the reason it went on for so long undetected is that the whole proprietary process for distributing the software was limited to so few people it was easy to miss. These kind of attacks happen in open source projects, too, they just get caught much faster.

That is the beauty of being able to see the code. You have the choice to build your own packages if you want, and you can examine code changes to your hearts content.

We host OpenNMS at Github. If you check out the code you could run something like:

git tag --list

to see a list of release tags. As I write this the latest released version of Horizon is 26.0.1. To see what changed from 26.0.0 I can run

git log --no-merges opennms-26.0.0-1 opennms-26.0.1-1

If you want, there is even a script to run a “release report” which will give you all of the Jira issues referenced between the two versions:

git-release-report opennms-26.0.0-1 opennms-26.0.1-1

While that doesn’t guarantee the lack of malicious code, it does put the control back into your hands and the hands of many others. If something did manage to slip in, I’m sure we’d catch it long before it got released to our users.

Security is not easy, and as with many hard things the burden is eased the more people who help out. In general open source software is just naturally better at this than proprietary software.

There are only a few people on this planet who have the knowledge to review every line of code on a modern computer and understand it, and that is with the most basic software installed. You have to trust someone and for my peace of mind nothing beats the open source community and the software they create.

Prodigal Customers

Growing up in the southern United States meant Sunday mornings were spent at Sunday School. One of the stories we would study was the Parable of the Prodigal Son. A man has two sons. The younger son asks for his inheritance in advance and he goes off and squanders it. When he returns, his father throws a big celebration to welcome him back.

I never really got the point of that story, as I always identified with the older, dutiful son, so it is surprising that it took working with OpenNMS for me to understand it.

We have great customers. Since we do little marketing, before we get a customer they have to first discover OpenNMS, then investigate it to see if it meets their needs, and only then do they contact us. It means that they are self-selecting, and without exception they are incredibly smart, physically beautiful and possessing of a wit so sharp they make Ginsu knives look dull. (grin)

The first company to ever buy an OpenNMS support subscription did so in December of 2001, and this year they renewed for the 17th time. It is a wonderful testament to the work of the team that they created something to inspire such a long commitment.

That said, we do lose a few customers each year. The first one I lost was a little heartbreaking. It was a hospital in Virginia, and when I called them to see if they would renew their support subscription they told me “no”. I was a little shocked, as I was unaware of any problems and they hadn’t opened tickets in awhile, and they told me that was the point. They loved OpenNMS but it “just worked” so they saw no value in getting support, they were still using it.

A more common case for us losing a customer is that our “internal champion” leaves. OpenNMS is a complex and powerful tool, and it does take awhile to climb the learning curve to see its full potential. If all of that knowledge is focused on one person, and that person leaves, their replacement can be overwhelmed and seek out something simpler, even if it is more expensive and less powerful.

I am alway saddened when this happens, but lately we’ve been experiencing what I’m calling “Prodigal Customers”. These are customers who leave and come back.

Cartoon by Chad Essley http://www.cartoonmonkey.com

I love them, and always want to slaughter (figuratively) the fattened calf to welcome them back.

It’s hard to explain, but while it is wonderful to have someone use something you’ve created for almost two decades straight, it is almost more rewarding to have someone go and try something else and discover it doesn’t stack up. Heck, I’d love it if all our customers could try out every possible option, because those that then chose OpenNMS for their solution would truly recognize what an awesome platform it can be.

Being 100% open source, OpenNMS does not have any way to “lock in” a particular customer. You can use it with our services or without, but you always have access to the latest code. Thus choosing to use OpenNMS is a validation of the work we’ve put into it, and whether you are a long time customer, a new customer, or a “prodigal” customer, your preference to use OpenNMS makes all the work to create it worthwhile.

Welcome to 2018

I love New Year’s. Not exactly the party on New Year’s Eve, as I tend to spend it as a quiet evening with friends, but the idea of starting over and starting fresh.

It is also a good time to reflect on the year past. While 2017 was pretty tumultuous for the world at large, for OpenNMS it was a pretty good year.

Our decision to split OpenNMS into two versions is still paying off. We did three major releases of Horizon (19, 20, and 21) as well as point releases every month there wasn’t a major release, and Meridian 2017 finally came out, although later than I would have liked. Horizon users get to experience rapid advancements in power and features while Meridian users can relax knowing their system is very stable and secure.

While it is hard to pick out the best features added in 2017, I’d have to go with OpenNMS Helm and the Minion.

Helm allows you to combine and manage multiple instances of OpenNMS from a Grafana dashboard.

OpenNMS Helm

The Minion is our foray into the whole “Internet of Things” space with an application that can be installed on a small device and used to send remotely collected data to a central OpenNMS instance. Minions have minimal configuration and can be configured redundantly, yet they have the ability to collect massive amounts of monitoring data. We’re very eager to see what novel uses our users come up with for the technology (we have one customer that is “Minion-only”, i.e. they do no monitoring or collection from the central OpenNMS instance at all and instead just put two Minions at each location).

As for the OpenNMS Group, the company behind OpenNMS, we experienced modest growth but still had a record year for gross revenue. What is more exciting is that net income was also a record and several hundred percent above last year, so we are going into 2018 well positioned in our Business Plan of “Spend less than you earn”.

2018 should be exciting. The OpenNMS Drift project brings telemetry (flow) data into OpenNMS, and we are working on some exciting features regarding correlation which will probably involve new machine learning technology.

As always, these features will be available as 100% free and open source software.

Personally, I added three new countries to my list, bringing the total number of countries I’ve been in to forty. I had a great time in Estonia and Latvia, and I really enjoyed my trip to Cuba.

One last thing. If you are reading this you are probably a user of OpenNMS. If so, thank you. We are a small but dedicated group of people creating this platform and often we don’t get much feedback on who uses it and what they like about it. The fact that people do find it useful makes it worthwhile, and we wouldn’t exist without our users and clients.

So, Happy New Year, and may 2018 exceed your wildest expectations.

Update on Expensify

I recently posted a rant on how a vendor we use, Expensify, appeared to be exposing confidential data to workers with the Amazon Mechanical Turk service. In response to the general outcry, they posted a detailed explanation on their blog.

It did little to change my mind.

So apparently what happened is that they used to use the Mechanical Turk from 2009 to 2012, so if you we a customer back then your information was disclosed to those third party workers. Then they stopped, supposedly using some other, similar, in-house system.

But, some genius there decided that the best way for certain customers to insure their receipts were truly private was to have them use the Mechanical Turk with their own staff. I covered that in my first post and it is so complex it hardly registers as a solution.

Of course, they decided to test this new “solution” starting the day before the American Thanksgiving holiday. This was done using receipts from “non-paying customers”. While we pay to use the service (not for much longer), if you were trying it out for free your receipts were exposed to Mechanical Turk workers. Heh, if you aren’t paying for the product you are the product. The post goes on to talk about the security of the Mechanical Turk service, which was surprising because they went on and on about how they didn’t use it.

What really angered me was this paragraph:

The company was away with our families and trying hard to be responsive, while also making the most of a rare opportunity to be with our loved ones. Accordingly, this vacuum of information provided by the company was filled with a variety of well-intentioned but inaccurate theories that generated a bunch of compounding, exaggerated fears. As a family-friendly business we try hard to separate work life from home life, and in this case that separation came at a substantial cost.

Well, boo hoo. If you truly cared about your employees you wouldn’t start a major beta test the day before a big holiday. I spent my holiday worrying about my employees’ personal data possibly being exposed through the Expensify service. Thanks for that.

What pisses me off the most is this condescending Silicon Valley speak that their lack of transparency is somehow our fault. That our fears are just “exaggerated”. When Ryan Schaffer posted on Quora that nothing personal is included on receipts, he demonstrated a tremendous lack of understanding about something on which he should be an expert. As they turn this new leaf and try to be more transparent, I noticed he deleted his answer from the Quora question.

Smells like a cover up to me.

Look, I know that being from North Carolina I can’t possibly understand all the nuances of the brain-heavy Valley, but if Expensify truly does have a “patented, award-winning” methodology for scanning receipts, why don’t they just make that available to their customers instead of using the Turk? This long-winded defense of the Turk seems like they are protesting too much. Something doesn’t make sense here.

I’ve told my folks to stop using SmartScan and that we would move away from Expensify at the end of the year. If you use, or are planning to use, Expensify you should deeply consider whether or not this is a company you want to associate with and if they will act in your best interests.

I decided the answer was “no”.

Dougie Stevenson – The Elvis of Network Management

David messaged me yesterday that Dougie Stevenson had died.

I hadn’t seen Dougie in person in a long time, but I’d kept up with him through the very networks he, in part, helped manage. While I had heard he wasn’t in the best of health, the news of his passing hit me harder than I expected.

I can’t remember the first time I met Dougie. I do remember it was always Dougie, rarely Doug and never Douglas. While most adults might drop such a nickname, it is a reflection on his almost childlike friendliness and good nature that he kept it. I do know that I was working at a company called Strategic Technologies at the time, so this would be the mid-1990s. I was working with tools like HP OpenView, and I’d often run into Dougie at OpenView Forum events. When he decided to take a job at Predictive Systems I followed him, even though it meant commuting to DC four to five days a week.

It was at Predictive that I got to see his genius at work. With his unassuming nature and down-to-earth mannerisms it was easy to miss the mind behind them, but when it came to seriously thinking about the problems of managing networks there were few who could match his penchant for great ideas. I used to refer to him as the “Elvis” of network management.

We were both commuters then. While he had lived in many places, he called Texas home as much as I do North Carolina. We were working on a large project for Qwest near the Ballston metro stop, and after work we’d often visit the nearby Pizzeria Uno. The wait staff loved to see Dougie, and would always laugh when he referred to the cheese quesadillas appetizer as “queasy-dillies”. This was back during the first Internet bubble, around 1999, and while many of us were working hard to make our fortune, Dougie never really cared that much for money. He used to joke it would all go to his ex-wives anyway. I know he had been married but we didn’t talk too much about that aspect of his life. He’d much rather talk about the hotrod pickup truck he was always working on when he had the time. I do remember he once walked away from a small fortune over principles – that was just the kind of person he was.

I can’t remember the last time I saw Dougie, but it could have been in Austin back in 2008. I have this really bad picture I took then:

Dougie and Me

Notice he has on his OpenNMS shirt. He never failed to promote our efforts to create a truly free and open source network management platform whenever he could.

As I’ve gotten older, I wish more for time than money. Between the business and the farm I’m kept so busy that I rarely get to spend as much time with the amazing people I know, and it would have been nice to see Dougie at least once more. In any case, a small part of him lives on in the hearts and minds of those who did know him.

Though it saddens me to say it, Elvis has left the building.

Expensify and Why I Hate the Cloud

Over the weekend I found out that Expensify, a service I use for my company, outsources a feature to Amazon’s Mechanical Turk service. Expensify handles the management of business expenses, which for a company like ours can be problematic as we do a lot of travel when deploying services. The issue is that the feature, the “smart scanning” of receipts, could potentially expose confidential data to third parties. As a user of Expensify, this bothers me.

Expensify touts “SmartScan” as:

As background, SmartScan is the patented, award-winning technology that underpins our “fire and forget” design for expense management. When you get a receipt, rather than stuffing it into your pocket to dread for later, just:

1. Take your phone out of your pocket
2. SmartScan the receipt
3. Put your phone back in your pocket

What they never told us is that if their “patented, award-winning technology” can’t read your receipt, they send it to the Mechanical Turk, which in turn presents it to a human being who will interpret the receipt manually. The thing is, we have no control over who will see that information, which could be confidential. For example, when I post a receipt for an airline ticket, it may include my record locater, ticket number and itinerary, all of which are sensitive.

This apparently never occurred to the folks at Expensify. Take this Quora answer from Ryan Schaffer, listed as Expensify Director of Marketing & Strategy:

Also, its worth mentioning, they don’t see anything that can personally identify you. They see a date, merchant, and amount. Receipts, by their very nature, are intended be thrown away and are explicitly non-sensitive. Anyone looking at a receipt isunable to tell if that receipt is from me, you, your neighbor, or someone on the other side of the world.

Wrong, wrong, wrong. It seems that Mr. Schaffer may limit his business expenses to the occasional coffee at Starbucks, but for the rest of us it is rarely that limited. For someone whose job is to perfect dealing with receipts, his view is pretty myopic.

For examples of what Expensify exposes, take a look at this tweet by Gary Pendergast.

Information Exposed by Expensify Tweet

It is also worth noting that it appears Expensify does its business on the Mechanical Turk as “Fluffy Cloud” instead of Expensify, which strikes me as a little disingenuous.

In a blog post this morning the company addressed this:

As you might imagine, doing this is easier said than done. Given the enormous scale and 24/7 nature of this task, we have agents positioned around the world to hand off this volume from timezone to timezone. Most of the US team is located in Ironwood, MI or Portland, OR (where we have offices and can train in person). Most of the international team is in Nepal or Honduras (where we work with a third-party provider to manage the on-site logistics). But regardless of the location, every single agent is bound by a confidentiality agreement, and subject to severe repercussions if that agreement is broken.

But if this were true, why are random people on Twitter announcing that they can see this data? Are they relying on the Amazon agreement with the people working as part of the Mechanical Turk? That doesn’t instill much confidence in me. But then in the same blog post they double down, and suggest that if you want extra security, you can just set up your own staff as part of the Mechanical Turk:

1. You hire a 24/7 team of human transcription agents.
         o For the fastest processing we suggest staffing three separate shifts — or daytime shifts in three different offices around the world. Otherwise your receipts might lag for many hours before getting processed.

2. They apply to Amazon Mechanical Turk for an account. Be aware that this is a surprisingly involved process, including:
         o The agent must sign up using their actual personal Amazon account. If your account doesn’t have an adequate history of purchases (each of which implies a successful credit card billing transaction and package delivery) or other activity, you will be rejected.
         o The agent must provide their full name, address, and bank account information for reimbursement. Amazon verifies this with a variety of techniques (eg, confirm that your IP is in the country you say you are, verify the bank account is owned by the name and address provided, full criminal background check), and if anything doesn’t add up, you will be rejected.
         o Rejection is final. It requires such an abundance of verifiable documentation (most notably being an active Amazon account with a long history) that you can’t just create a new account and try again.
         o There is no apparent appeals process. Accordingly, I would recommend confirming before hiring that the candidate can pass Amazon Mechanical Turk’s many strict controls because we have no ability to override their judgement.

3. You notify us of the “workerID” of each of your authorized agents.
         o Though you are not obligated to share your staff’s identity with us directly, your staff will still be obligated to follow the Expensify terms of services. Failure to comply with our terms will result in an appropriate response, starting with immediate banning by our automated systems, ranging up to our legal team subpoenaing you (or failing that, Amazon) for the identity of the agent to press charges directly.

4. We will create a “Qualification” for your “Human Intelligence Tasks” (HITs) that ensures only your agents will see your receipts.

5. Your staff will use the Amazon Mechanical Turk interface to discover and process your employee’s receipts.

That’s the solution? This is what passes for security at Expensify? Hire three shifts of employees all using verified personal Amazon accounts and then you can be sure your confidential data is kept confidential?

Wouldn’t it just be easier to create a small webapp that would present receipts to people in a company directly without going through the Mechanical Turk? Heck, why not just bounce it back out to you – it isn’t that great of a chore.

Plus, basically, if you don’t do this Expensify is saying they can’t keep your information secure.

This is what frustrates me the most about “the Cloud”. Everyone is in such a rush to deploy solutions that they just don’t think about security. Hey, it’s only receipts, right? Look what I was able to find out with just a discarded boarding pass – receipts can have much more information. And this is from a company that is supposed to be focused on dealing with expenses.

I demand two things from companies I trust with our information in the cloud: security and transparency. It looks like Expensify has neither.

I will be moving us away from Expensify. If you know of any decent solutions, let me know. Xpenditure looks pretty good, and since they are based in the EU perhaps they understand privacy a little better than they do in San Francisco.

2017 Australian Network Operators Group Conference

Back in June I was chatting with “mobius” about all things OpenNMS. He lives and works in Perth, Australia, and suggested that I do a presentation at AusNOG, the Australian Network Operators Group.

One of the things we struggle with at OpenNMS is figuring out how to make people aware it exists. My rather biased opinion is that it is awesome, but a lot of people have never heard of it. To help with that I used to attend a lot of free and open source conferences, but we’ve found out over the years that our user base tends to be more along the lines of large enterprises and network operators that might not be represented at such shows.

Imagine my surprise when I found out that there were a whole slew of NOGs, network operator groups, around the world. It seems to me that people who attend these conferences have a more immediate need for OpenNMS, and with that in mind I submitted a talk to AusNOG. I was very happy it was selected, not in the least because I would get to return to Australia for a third time.

AusNOG David Hughes

I wasn’t sure what to expect, but was pleasantly surprised. The event was extremely well organized and I really liked the format. Many conferences as they become successful respond by adding multiple tracks. This can be useful if the tracks are easy to delineate, but often you can get “track bloat” where the attendees get overwhelmed with choice and as a presenter you can end up with a nearly empty room if you are scheduled against a popular speaker. At AusNOG there is only one track of somewhat short, highly curated talks that results in a very informative conference without the stress of trying to determine the best set of talks to attend.

AusNOG Program

(Note: Visit the “programme” site and click on talk titles to download the presentations)

The venue was very nice as well. Held at the Langham Hotel, the conference took place in a ballroom that held the 200+ people with a lobby out front for socializing and a few sponsor booths. The program consisted of 90 minutes of presentations separated by a break. They alternated sets of three 30 minute talks with two 45 minutes talks. I found all of the presentations interesting, but I have to admit that I spent a lot of time looking up unfamiliar acronyms. As network operators Autonomous System (AS) numbers were thrown around in much the same way SNMP Private Enterprise Numbers are shared among network management geeks. Australia is also in the process of implementing a nationwide National Broadband Network (nbn™) to provide common infrastructure across the country, so of course that was the focus of a number of talks.

In the middle of each day we broke for lunch which was pretty amazing. The Langham restaurant had a sushi section, a section for Indian food, a large buffet of your standard meat and veg, and at least three dessert sections: one with “healthy” fruit and cheese, another focused on ice cream and a chocolate fountain, and a large case full of amazing pastries and other desserts. All with a view out over the Yarra river.

AusNOG Yarra River

I really liked the format of AugNOG and suggest we adopt it for the next OpenNMS conference. For those few talks that were either over my head or not really of interest, they were over pretty quickly, but I found myself enjoying most of them. I thought it was interesting that concepts we usually equate with the managing servers are being adopted on the network side. One talk discussed topics such as running switch software in containers, while another discussed using Ansible and Salt to manage the configuration of network gear.

AusNOG Runing Switch Software in Containers

I was happy to see that my talk wasn’t the only one that focused on open source. Back fifteen years ago getting large companies to adopt an open source solution was still in the evangelical stage, but now it is pretty much standard. Even Facebook presented a talk on their open source NetNORAD project for monitoring using a distributed system to measure latency and packet loss.

I did have a few favorite talks. In “The Future Is Up in the Sky” Jon Brewer discussed satellite Internet. As someone who suffered for years with a satellite network connection, it was interesting to learn what is being done in this area. I used a system with a satellite in geosynchronous orbit which, while it worked, ended up with latency on the order of a second in real-world use. It turns out that there are a number of solutions using shorter distances with satellites in low earth or high earth orbit. While they present their own challenges, it is still the most promising way to get network access to remote areas.

Another talk by Mark Nottingham discussed issues associated with the increased use of encrypted protocols and the challenges they create for network operators. While the civil libertarian in me applauds anything that makes it harder for surveillance to track users, as a network monitoring guy this can make it more difficult to track down the cause of network issues.

And this will become even more important as the network changes with the adoption of Internet of Things (IoT) devices. Another good talk discussed the issue of IoT security. Even today the main consensus is that you protect weakly secured devices with a firewall, but a number of new exploits leverage infected systems within the firewall for DDoS attacks.

AusNOG Internet of Things Vulnerabilities

I think my own talk went well, it was hard to squeeze a good introduction to OpenNMS into 30 minutes. I did manage it – 30 minutes on the nose – but it didn’t leave time for questions. As a speaker I really liked the feedback the conference provided in the form of a rather long report showing what the attendees thought of the talk, complete with cool graphs.

AusNOG Speaker Response

I really enjoyed this conference, both as an attendee and a speaker. While I hope to speak to more NOGs I would much rather encourage OpenNMS users who are happy with the project to submit real-world talks on how they use the platform to their local tech groups. I think it tells a much stronger story to have someone a little less biased than myself talking about OpenNMS, and plus you get to visit cool conferences like AusNOG.

Freedom Feud

My official title at OpenNMS is CEO, but I’ve worn several hats in the last 12+ years, including accountant, receptionist, HR manager and janitor. Now I get to add record producer to the list.

I guess it all started back in 2012. OpenNMS was doing pretty well and I wanted us to give a little something back to the community. As a fan of MC Frontalot I came across his FAQ and found out that you could actually book him for things like conferences, kids parties, bar mitzvahs and the like. We were sponsoring the Ohio Linuxfest (by the way, I’m a keynote speaker there this year along with the ever amazing Karen Sandler) and I decided to see if I could book him to play a show. Turns out he is pretty affordable (for contrast, Henry Rollins starts out at $10,000 per event, which isn’t unreasonable but doesn’t count as affordable for us just yet). I booked him to play a solo gig and finally got to meet the man. He did a great show, everyone seemed to enjoy it, and we became friends of a sort.

While Front is very much a nerd, he had not had much exposure to free software. A lot of musicians rely on Windows-based software to create their music (Front mainly uses Reaper and Professor Kliq is such an Ableton fan he has their logo tattooed on his wrists) and thus they aren’t used to using open source. The OLF event went so well I hired him a few more times, and I think it was at SCaLE when I suggested he write a free software song. His reply was, well, why don’t you commission one.

Front is talent for hire. He did a couple of tracks for New Relic, “Nerd Life” and “Small Data“, and while we don’t have anywhere near the budget of that company we felt that free software deserved to be examined under his lyrical microscope, so we started the process.

Note that this was a couple of years ago, back in 2015, so it took awhile. Front and I had a number of conversations about free software and I started him on his education. I pointed him to the works of Richard Stallman and Eric S. Raymond (notably The Cathedral and the Bazaar), as well as Lawrence Lessig and organizations such as the Free Software Foundation (FSF) and the Electronic Frontier Foundation (EFF). We also talked about the classic “free as in beer” vs. “free as in freedom” confusion that arises out of the term “free software”, which ended up forming the basis of the song.

Later in 2015 I wanted to do something special for the OSCON show in Portland. This time I decided to hire Front along with his band. In the previous shows he’d done for us it was him and “DJ CPU” providing the music, and while those were great shows I was unprepared for the “live band” experience. It took it to another level. During that show Front performed a bit of the song, but it wasn’t until last year’s All Things Open that the whole song was played for the first time (again with the band, since, awesome).

With the song almost complete we now how to figure out how to present it to the world. I wanted a video, so I decided to turn to animator Chad Essley. Chad had done the video for “Shudders” off of Question Bedtime and we had gotten to know each other through a fund raising promotion he did for the EFF where we sponsored adding OpenNMS references to that video. I felt he could do a good job with it, since he is both talented and he gets the subject matter.

Now when I said “record producer” above I basically meant I signed the checks, but it was cool watching artists such as Front and Chad work out even small details when it came to the video. Animation can take a really long time, so we debated on combining it with some live action, etc., to both speed up the process and reduce the cost. It was at this point that I was introduced to the concept of a “lyric video”.

Ed Sheeran had just come out with a new song, and in order to promote it as quickly as possible he released a video that pretty much consisted of just the song’s lyrics. While at this point in our process we had a portion of the animation completed, I thought that adding the lyrics to it would both speed things up as well as improve the experience, as Front’s rhymes on this track are some of the tightest he’s ever written. The end result is both a video that is fun to watch as well as one that gets the message across in an entertaining manner.

It seems to have been well received, and as I write this it has had over 10,000 views and mostly positive comments, and we got a shout out on Boing Boing.

Speaking of lyrics, the phrase that has gotten the most comments is the line “Pull down capitalism till it’s rubble and chain”. When asked about it, Front commented:

I thought it was funny to equate Free Software with that dirty commie yearning for collectivized well-being and shared responsibility! Those two realms of thought are not directly in line with each other as far as I can tell. But I imagined the kind of business dude who is reflexively distrustful of free software, and I pictured him saying, “you mean… LIKE COMMUNISM??” Thought I’d give him a little dig at the end there.

I can’t remember if I shared my story with Front when we were first talking about creating this song, but when I got started with OpenNMS I was called a communist a number of times on various message boards. It confused me, since it came mainly from people who made their living as network management consultants. I was thinking, hey, here is a tool that lets you provide better solutions for your customers while showcasing your unique talents via your ability to deliver them, and that’s communism? Sounds like good business to me. But there is still that element of “anti-capitalism” associated in free software (I blame the phrase “so you can help your neighbor” in the Free Software Definition, but that’s just me).

The whole process was a lot more work than I thought it would be, but I’m very pleased with the result. Check out the video as there are a lot of in-jokes and Easter eggs, and I’ve been told that “floppy head Lawrence Lessig” was met with approval by the man himself.

Oh, in the spirit of free software, Front has published the song under the Creative Commons (CC BY-SA) license, and I am hoping to see a few cool remixes come out of this. I’ve reached out to both Professor Kliq and Raccoon Fink and if they find time to play with it, I’ll be sure to let you know (and let me know if you find some out there). Front is working on a new album tentatively entitled The Internet Sucks so maybe this track will make the cut.

When Not To Start an Open Source Company

Over the weekend, Chris Aniszczyk posted a link on Twitter to a very interesting article by Matt Klein about his decision not to start an open source company around his project, Envoy. I thought it raised a number of interesting points worth a few comments.

First off, Matt works for Lyft, which, in case you haven’t heard of it, is Uber without the moral decay. I abandoned Uber some time ago, despite being an early adopter, and I’ve been very happy with Lyft. One of the main differences is that Lyft allows you to tip your driver, which I almost always do with few exceptions. The fact that Lyft is able to keep and motivate people like Matt speaks volumes for their corporate culture.

It also demonstrates a wonderful trend of commercial companies starting and maintaining open source projects. I’ve been working with open source for almost two decades and I can remember when any software developed at a company was considered confidential. To this day there are a number of vendors who consider their SNMP MIB files (which, I should point out, are really only useful to people who have purchased their products) proprietary information. Companies like Lyft, Paypal and Facebook, none of which would self-identify as open source companies, have gained a lot of value for little cost by making the tools they use open source.

When talking about open source for the enterprise, I often talk about the fact that it is the processes that a company uses to serve its customers that make it unique and define its value, not the tools used by the company. So often with commercial software you have to change those process to fit how the application thinks you should work, and in the process you lose some part of what makes you special to your customers. With open source you can fit the application to those processes. It is how you use the tools and not the tools themselves that is important, and so there is a lot to gain and little to lose by making them open source.

Getting back to Matt’s article, he is a project maintainer for Envoy, which is a “high performance C++ distributed proxy and communication bus designed for large service oriented architectures.” While I don’t consider myself a coder so I don’t claim to fully understand the its advantages, I do recognize enough buzzwords in that sentence to know that it would attract some attention from investors, and Matt was approached about leaving Lyft to start a commercial business around Envoy. He decided not to, and as I read his article about his decision I realized I’d found a kindred soul, someone who was more interested in creating something of value that would last versus making a quick buck.

He had me with this paragraph:

In my opinion, the best opportunity to commercialize OSS lies with projects that can be easily turned into SaaS products. Ultimately, even if software is completely open, many customers are happy to pay for a turnkey solution that “just works” and has a defined SLA with 24/7 monitoring and support. In some sense, customers pay for the operational expertise that comes from deeply understanding and running the software, versus the software itself.

Amen.

I’ve been making a living on open source for 15 years now working with OpenNMS, and I’ve spent a lot of time thinking about business models. We started out with the “service and support” model, which kept the doors open but limited growth. Then our clients started asking us for features, so we added custom development, which was time intensive but allowed us to finance OpenNMS features which attracted even more customers as the platform became more powerful. When we hit the problem of trying to balance the “release early, release often” philosophy of open source with the need for stability, we adopted the Red Hat model of splitting our application into a feature-rich, rapidly developed release (which we call Horizon™, similar to Fedora) and a more stable, subscription-based release that may lag in features but is better suited production environments (which we call Meridian®, similar to RHEL). But ultimately we came to the decision that what we really wanted to do was to offer OpenNMS as a service.

One company that inspired that decision was Automattic, maintainers of WordPress. I don’t think I know of a more powerful piece of software that is easier to install. They have a famous “5 Minute Install” that is quite simple. First, you drop the software into the webroot of your web server of choice. Next, you create a database account on your database of choice with certain permissions. Then you navigate to a web page and follow the prompts.

However, for a lot of people, terms like “webroot” are gibberish, and even with WordPress you still need some minimal database skills to maintain it. So Automattic offers up WordPress as a service. For a small monthly fee they’ll do everything for you, and this has generated revenues on the order of tens of millions of dollars per year.

OpenNMS is way more complicated, thus the value of a hosted version should be greater. In order to do so we needed some way to access the client’s network in a secure fashion, so with Horizon 20 we introduced the Minion. The Minion software allows for OpenNMS functionality to be distributed. It is built on the Karaf container, so once installed all of its features can be remotely managed. For smaller networks, the Minion can be sold as an appliance and talk to a hosted version of OpenNMS. It can bring a complex and powerful tool like OpenNMS into the hands of the masses.

For larger companies it solves issues of scale as Minions can be deployed to cover even the largest networks (our goal is IoT scale). We’ve had them in production at one client for months now handling over 2 million events an hour. That translates to around 555 events per second, although the system itself can handle over 10,000 events per second so they have room to grow. If they ever hit that limit, we can simply add more Minions. They have the option of hosting all of OpenNMS in their own data center, or they could choose a hybrid model where some of the functionality is outsourced.

For pretty much the first time in the history of OpenNMS, we are seriously and actively seeking investment. There are a number of companies entering this space who have raised enormous amounts of money, and we think we can be competitive for far less money and provide a better solution. Plus, also for the first time in the history of OpenNMS, we have a reason to make it easier to use versus spending all of our resources making it more powerful.

Matt talks about investment in his post (remember Matt? As usual, I’ve made this all about me. Meeee!) It was actually his stories about dealing with investors that prompted me to write this. As Envoy started to get some traction, investors wanted him to leave and start a company. He writes:

Over the last few months I’ve been told by several investors that no OSS has become ubiquitous without having explicit commercial backing. I think this is false and is situation dependent. If anything, I would argue that if I were to leave Lyft now and start a platform company around Envoy, it will decrease the chance of Envoy becoming ubiquitous, primarily because it would negate all of the reasons laid out above.

That first sentence is interesting, since “ubiquitous” and “commercial” are a little vague. I would make the claim that the Apache web server was ubiquitous until its success spawned NGINX, and it was backed by the Apache Software Foundation which is a non-profit. Is a foundation “commercial”? The idea that for a project to become successful it needs a number of people to spend a lot of time working on it seems obvious, and the best way to achieve that is to pay those people to work on it.

He goes on to write:

It took me a lot of time to ultimately understand the previous simple point. Investors are extremely persuasive. They capitalize on “fear of missing out.” However, it’s important to realize that the opportunity cost is hugely mismatched between investor and company.

When he writes “investors” above I believe he means specifically venture capitalists. We’ve talked with a few VCs in the past and I can remember the almost “strong arm” tactics they used. If I hear “a rising tide lifts all boats” one more time, I might have to hit somebody. I’m not saying that all VCs are the same, but many of them come across as gamblers and not investors. I’m risk friendly but I don’t gamble. I’m heavily invested in wanting to build something with OpenNMS that outlasts me (it is already much bigger than me as the team I work with has way more to do with its success than I do) and I don’t want to gamble with it.

I do hope that there are some investors out there that can appreciate that aspect of our company as well as the fact that we’re profitable, have mature products and wonderful customers. Perhaps private equity or perhaps another company that shares our vision and wants to advance the project through acquisition. In any case we’re looking for them.

When I was a young man, old guys like I am now would tell me “work on something you love, not just for the money”. I always dismissed it with the thought that with enough money I can buy love. When you immerse yourself in something as personal as an open source project for ten to twelve hours a day, year after year, you really do have to love it and the satisfaction you get just can’t be bought. Matt’s thoughts are similar:

Ultimately, on a personal level I’m just having too much fun solving tough computer science problems at large scale at Lyft and building a community around Envoy. The bar to do something different is therefore extremely high, and it took a long time to realize that it’s perfectly OK to accept that and keep going down the existing path that I’m on. On another level, leaving now to start a company would feel very much like not following through on my original goal of open sourcing Envoy; the industry desperately needs a high quality and community-driven solution to microservice networking. Follow-through is something I take very seriously.

With that attitude the success of Envoy is almost assured.

Why the FCC’s Title II is so Important (Spectrum Rant)

Here is a rant about Time Warner/Charter/Spectrum or whatever the heck they call themselves these days. It illustrates how this large company can have a huge negative impact on a small business, and why treating Internet providers as common carriers is so important.

Our company wouldn’t exist without the Internet. Outside of the fact that our products are mainly used to monitor Internet resources, we host a number of servers from our office and about half of the staff works remotely so we rely on the Internet to communicate and coordinate.

Back in 2012 I contracted with Time Warner to provide Internet access to our office. We had fiber to the building and while our service was considerably more expensive than coax, I liked the fact that it was symmetrical and expandable. We started of with 20 Mbps but soon increased that to 50 Mbps. Over five years we only had one outage, due to a misconfiguration of our Customer Premise Equipment (CPE), and they corrected it within 20 minutes. I love the fact that when you called in the person who answered the phone understood terms like “duplex” and they were always very helpful.

Note the scenario: happy customer who is happy paying a premium for enterprise-level service.

Now let me tell you why all that goodwill has gone away.

Earlier this year we decided to move our office from Pittsboro, NC to Apex, NC. The first thing I did was contact Time Warner (well, Charter at the time) to insure that they could provide fiber to the new location. They said they could, although it would take 45 to 60 days. As our new office space needed to be completed, we were targeting an April 1st move in date anyway, so on February 15th I placed the order for the new service. At best, it would be available on the 1st and at worst it would be ready by the 15th. We told the old landlord we’d be out by April 30th just in case and to give us more time to move.

Finally, Spectrum doubled our speed and cut the price in half. I was feeling pretty good about the whole thing.

The feeling didn’t last.

As we got closer to April, things started to go wrong, most of it due to the fact that Spectrum is now such a behemoth that they have no idea what they are doing. In order to get fiber into our new building, they needed what is called a “Right of Entry”. They sent it to our landlord who promptly completed the form and sent it back. However, that person didn’t let the project manager know the form had been received, so he did absolutely nothing. Ten days (!) later I get a note that our build out had been suspended because of the lack of the ROE form. A form, I should point out, that was sent to them, twice.

At the end of March I’m told that our new date is May 11th. I’m unhappy – due to their poor processes I now have a new office that I can’t use for six weeks (remember, we took possession and started paying rent on April 1st). We also had to be out of the old office by the end of April. Luckily I work with a great team that is able to be productive when working from home, so I decided to suck it up and live with it.

On April 12th I get an update – the new date for the end of construction is now May 15th due to processes within Spectrum taking too long to finalize the work with a contractor. Now the actual date we’ll have Internet has been pushed out to the week of May 29th.

I am livid. By this point I’m ready to switch to the other option, AT&T. Unfortunately, they also need 45 to 60 days for service installation so I realize at this point I’m stuck with Spectrum.

I ask my salesperson for options and he suggests we get coax installed for a month (for a fee, of course). Since our office is right next to a large housing development they can get coax in the following week. I sign off on it.

It didn’t happen. When May arrived some of us started working in the new office mooching off the neighbor’s Wi-Fi from AT&T (with permission of course). I ended up traveling for a couple of weeks so I completely forgot about the coax option (it’s not like Spectrum was keeping me updated on anything – I’d have to reach out to them for an update). I did get a note on May 10th that all construction had been completed for the fiber and another note on May 18th that our new install date was June 2nd.

(sigh)

So, 45 days late, we have a firm install date. Wonderful.

Imagine how I felt when on the 24th of May I received a note that more construction was needed and that it would be pushed out another 30 days at least. When I get extremely angry I refer to it as going “non-linear” as that how fast my blood pressure rises. As I was ranting to pretty much everyone I’d ever interacted with at Spectrum it dawned on me that this could be for the coax order. Turns out that was the case. Apparently our crack project manager on the coax side decided to route our service from a point several miles away instead of from the one nearly across the street. This is why it was delayed and why the construction was needed. By this time we are about a week out from having fiber so I canceled the order. I did get a very apologetic call from the coax salesperson which I appreciated (under Spectrum, fiber [Enterprise] is handled by one sales team and coax [Business] is handled by another), and I made it clear that I’d be okay with everything as long as the fiber was delivered as promised on the 2nd.

It was. Around noon on June 2nd we had our 100 Mbps service and on the 3rd we moved all of our devices from the old office in Pittsboro to the new one in Apex. I informed my salesperson that they could disconnect the old service and despite all of the problems, I was happy with the new service.

So the whole process cost me two months rent and a few years off my life, but it was finally over.

Not so fast – the other shoe fell today.

I get an e-mail that I need to confirm my disconnect request. That didn’t bother me, in fact I appreciated it, but what did bother me was an additional note that it would be done within 30 days. When I replied I asked for clarification – would I be *paying* for the service I wasn’t using until they could disconnect it? The answer was “yes”.

I experienced a new word – apoplectic.

Due to the fact that the bureaucracy behind the new merged Spectrum company is so bad, I’m out nearly ten thousand dollars. That is the real money – it’s probably cost us twice that again in lost productivity from lack of network access and dealing with them throughout this process. We’re not one of those companies that is too big to fail so this really impacts us negatively. Had it been explained to me that I’d have to pay for the service until it was disconnected, I would have put the disconnect order in a month ago, but then had I used the date I was originally promised, our servers would have been off-line for over a month. That would have been catastrophic to our company.

Finally, I’ve gone from a happy customer to an extremely pissed off one who will be actively looking for options. Based on my experience I would suggest any business looking for network access look elsewhere.

Access to the Internet has become as important as other utilities such as electricity, water and sewer and just like those utilities it needs to be regulated as one. This is why the decision by the new industry-picked head of the FCC to reverse the decision to classify Internet access under Title II as a “common carrier” is so devastating to businesses like mine. Our company is small, yet we put millions of dollars into the local economy each year. You multiply that by the number of other small businesses and it can have a great impact to any community. Barriers put up by companies like Spectrum demonstrate that they can’t self-regulate and the government needs to take a firmer hand (and this is coming from a left-leaning libertarian).

I will be protesting that final bill for Internet access and I would welcome any advice on how to deal with a company like Spectrum. Let’s hope that there is a change soon so that other businesses can focus on creating value and not have to deal with the crap we had to endure.

I’m not holding my breath.