open source

OpenNMS Is Once Again on FLOSS Weekly

Posted on by Tarus

Way back in 2006 I was invited to be on one of the first FLOSS Weekly shows. That was when it was hosted by Chris Dibona and Leo Laporte. Now it is run by the very capable Randal Swartz, and I was excited to be invited back, ten years later. It was also fun to meet Jonathan Bennett, his co-host, for the first time.

Jeff Gehlbach joined me to chat about OpenNMS and all things FLOSS, and I even thought he got a word or two in edgewise. Like FLOSS Weekly, I think our major achievement is that we are still here and still going strong (grin). The only complaint I could have is that this was episode 418 and I was originally on episode 15 so it would have been cooler to be on three shows ago to make it an even 400, but I’m OCD like that.

FLOSS Weekly

One thing I love about free (libre) and open source software is that it is self-selecting. People choose to use it, and thus there tend to be certain things we all hold in common that makes meeting others involved in FLOSS like immediately making a new friend. Chatting with Randal and Jonathan was more like catching up with old friends, although I’d never talked with them before. I look forward to this as the beginning of a beautiful friendship.

Anyone who has had the misfortune of listening to me drone on about OpenNMS in the past will here a number of “bingo” stories in this show, but we do touch on some new ideas and I think it went really well. Please check it out and let me know what you think.

Review: Copperhead OS

Posted on by Tarus

A few weeks ago I found an article in my news feed about a Tor phone, and it introduced me to Copperhead OS. This is an extremely hardened version of the Android Open Source Project (AOSP) designed for both security and privacy. It has become my default mobile OS so I thought I’d write about my experiences with it.

TL;DR: Copperhead OS is not for everyone. Due to its focus on security is it not easy to install any software that relies on Google Services, which is quite a bit. But if you are concerned with security and privacy, it offers a very stable and up to date operating system. The downside is that I am not able to totally divorce myself from Google, so I’ve taken to carrying two phones: one with Copperhead and one with stock Android for my “Googly” things. What we really need is a way to run a hypervisor on mobile device hardware. That way I could put all of my personal stuff on a Copperhead and the stuff I want to share with Google in a VM.

I pride myself to the point of being somewhat smug about the fact that I use free software for most of my technology needs, or so I thought. My desktops, laptop, servers, router, DVR and even my weather station all use free and open source software, and I run OmniROM (an AOSP implementation) on my phone. I also “sandbox” my Google stuff – I only use Chrome for accessing Google web apps and I keep everything else separate (no sharing of my contacts and calendar, for example). So, I was unpleasantly surprised at how much I relied on proprietary software for my handy (short for “hand terminal” or what most people call a “mobile phone”, but I rarely use the “phone” features of it so it seems like a misnomer).

But first a little back story. I was sitting on the toilet playing on my mobile device (“playing on my handy” seemed a little rude here) when I came across a page that showed me all of the stuff Google was tracking about my mobile usage. It was a lot, and let’s just say any bathroom issues I was having were promptly solved. They were tracking every call and text I made, which apps I opened, as well as my location. I knew about the last one since I do play games like Ingress and Pokémon Go that track you, but the others surprised me. I was able to turn those off (supposedly) but it was still a bit shocking.

Of course, I had “opted in” to all of that when I signed in to my handy for the first time. When you allow Google to backup your device data, you allow them to record your passwords and call history.

Google Backup Terms

If you opt in to help “improve your Android experience”, you allow them to track your app usage.

Google App Terms

And most importantly, by using your Google account you allow them to install software automatically (i.e. without your explicit permission).

Google Upgrade Terms

Note that this was on a phone running OmniROM, and not stock Google, but it still looks like you have to give Google a lot of control over your handy if you want to use a Google account.

Copperhead OS is extremely focused on security, which implies the ability to audit as much software on the device as possible, as well as to control when and what gets updated. This lead them to remove Google Play Services from the ROM entirely. Instead, they set up F-Droid as the trusted repository. All the software in F-Droid is open source, and in fact all of the binaries are built by the F-Droid team and not the developer. Now, of course, someone on that team could be compromised and put malicious software into the repo, but you’ve got to trust somebody or you will spend your entire life doing code reviews and compiling.

Copperhead only runs on a small subset of devices: the Nexus 6P, the Nexus 5X and the Nexus 9 WiFi edition. This is because they support secure boot which prevents malicious code from modifying the operating system. Now, I happened to have a 6P, so I figured I would try it out.

The first hurdle was understanding their terminology. On the download page they refer to a “factory” image, which I initially took to mean the original stock image from Google. What they mean is an image that you can use for a base install. If you flash your handy as often as I do, you have probably come across the process for restoring it to stock. You install the Android SDK and then download a “factory” image from Google. You then expand it (after checking the hash, of course) and run a “flash-all” script. This will replace all the data on your device, including a custom recovery like TWRP, and you’ll be ready to run Copperhead. Note that I left off some steps such as unlocking and then re-locking the bootloader, but their instructions are easy to follow.

The first thing you notice is that there isn’t the usual “set up your Google account” steps, because, of course, you can’t use your Google account on Copperhead. Outside of missing Google Apps, the device has a very stock Android feel, including the immovable search bar and the default desktop background.

This is when reality began to set in as I started to realize exactly how much proprietary software I used to make my handy useful.

The first app I needed to install was the Nova Launcher. This is a great Launcher replacement that gives you a tremendous amount of control over the desktop. I looked around F-Droid for replacement launchers, and they either didn’t do what I wanted them to do, or they haven’t been updated in a couple of years.

Then it dawned on me – why don’t I just copy over the apk?

When you install a package from Google Play, it usually gets copied into the /data/apps directory. Using the adb shell and the adb pull commands from the SDK, I was able to grab the Nova Launcher software off of my Nexus 6 (which was running OmniROM) and copy it over to the 6P. Using the very awesome Amaze file explorer, you just navigate to the apk and open it. Now, of course, since this file didn’t come from a trusted repository you have to go under Security and turn off the “trusted sources” option (and be sure to turn it back on when you are done). I was very happy to see that it runs just fine without Google Services, and I was able to get rid of the search bar and make other tweaks.

Then I focused on installing the open source apps I do use, such as K-9 Mail and Wikipedia, both of which exist in F-Droid. I had been using the MX Player app for watching videos, pretty much out of habit, but it was easy to replace with the VLC app from F-Droid.

I really like the Poweramp music player, with the exception that it periodically checks in with the Play store to make sure your license is valid. Unfortunately, this has happened to me twice when I was in an airplane over the ocean, and the lack of network access meant I couldn’t listen to music. I was eager to replace it, but the default Music app that ships with Copperhead is kind of lame. It does a good job playing music, but the interface is hard to navigate. The “black on gray” color scheme is very hard to read.

Default Music Player Screenshot

So I replaced it with the entirely capable Timber app from F-Droid.

Timber Music Player Screenshot

Another thing I needed to replace was Feedly. I’m old, so I still get most of my news directly from websites via RSS feeds and not social media. I used to use Google Reader, and when that went away I switched to Feedly. It worked fine, but I bristled at the fact that it tracked my reading habits. Next to each article would be a number representing the number of people who clicked on it to read it, so at a minimum they were tracking that. I investigated a couple of open source replacements when I was pleasantly surprised to discover that Nextcloud has a built in News service. We have had a really good experience with Nextcloud over the last couple of months, and it was pretty easy to add the news service to our instance. Using OPML I was able to export my numerous feeds from Feedly into Nextcloud, and that was probably the easiest part of this transition. On the handy I used an F-Droid app called OCReader which works well.

There were still some things I was missing. For example, when I travel overseas I keep in touch with my bride using Skype (which is way cheaper than using the phone) so I wanted to have Skype on this device. It turns out that it is in the Amazon App Store, so I installed that and was able to get things like Skype and the eBay and IMDB apps (as well as Bridge Baron, which I like a lot). Note that you still have to allow unknown sources since the Amazon repository is not trusted, and remember to set it back when you are done.

This still left a handful of apps I wanted, and based on my success with the Nova Launcher I just tried to install them from apks. Surprisingly, most of them worked, although a couple would complain about Google Services being missing. I think background notifications is the main reason they use Google Services, so if you can live without that you can get by just fine.

One app that wouldn’t work was Signal, which was very surprising since they seem to be focused on privacy and security. Instead, the default messenger is an app called Silence, which is a Signal fork. It works well, but it isn’t in the Play store (at least in the US due to a silly trademark issue that hasn’t been fixed) and no one I know uses it so it kind of defeats the purpose of secure messaging. Luckily, I discovered that the Copperhead gang has published their own fork called Noise, which removes the Googly bits but still works with the rest of the Signal infrastructure, so I have been using it as my default client with no issues. Note that it is in the F-Droid app but doesn’t show up on the F-Droid website for some reason.

For other apps such as Google+ and Yelp, I rediscovered the world wide web. Yes, browsers still work, and the web pages for these sites are pretty close to matching the functionality of the native app.

There are still some things for which there is no open source replacement: Google Maps, for example. Yes, I know, by using Google Maps I am sharing my location with Google, but the traffic data is just so good that it has saved literally hours of my life by directing me around accidents and other traffic jams. OpenStreetMap is okay and works great offline, but it doesn’t know where the OpenNMS office is located (I need to fix that) and without traffic it is a lot less useful. There is also the fact that I do like to play games like Ingress and Pokémon Go, and I have some movies and other content on Google servers.

I also lost Android Wear. I really enjoy my LG Urbane but it won’t work without Google Services. I have been playing with AsteroidOS which shows a lot of promise, but it isn’t quite there yet.

Note that Compass by OpenNMS is not yet available in F-Droid. We use Apache Cordova to build it and that is not (yet) supported by the F-Droid team. We do post the apks on Github.

To deal with my desire for privacy and my desire to use some Google software, I decided to carry two phones.

On the Nexus 6P I run Copperhead and it has all of my personal stuff on it: calendar, contacts, e-mail, etc. On the Nexus 6 I am running stock Google with all my Googly bits, including maps. I still lock down what I share with Google, but I feel a lot more confident that I won’t accidentally sync the rest of my life with them.

It sucks carrying two phones. With the processors and memory in modern devices I’m surprised that no one has come up with a hypervisor technology that would let me run Copperhead as my base OS and stock Google in a VM. Well, not really surprised since there isn’t a commercial motivation for it. Apple doesn’t have a reason to let other software on its products, and Google would be shooting itself in the foot since its business model involves collecting data on everything. I do think it will happen, however. The use case involves corporations, especially those involved in privacy sensitive fields such as health care. Wouldn’t it be cool to have a locked down “business” VM that is separate from a “personal” VM with your Facebook, games and private stuff on it.

As for the Copperhead experience itself, it is pretty solid. I had a couple of issues where DNS would stop working, but those seem to have been resolved, and lately it has been rock solid except for one instance when I lost cellular data. I tried reseting the APN but that didn’t help, but after a reboot it started working again. Odd. Overall is it probably the most stable ROM I’ve run, but part of that could be due to how vanilla it is.

Copperhead is mainly concerned with security and not extending the Android experience. For example, one feature I love about the OmniROM version of the Alarm app is the ability to set an action on “shake”. For example, I set it to “shake to dismiss” so when my alarm goes off I can just reach over, shake the phone, and go back to bed. That is missing from the stock ROM (but included in AOSP) and thus it is missing from Copperhead. The upside is that Copperhead is extremely fast with updates, especially security updates.

The biggest shortcoming is the keyboard. I’ve grown used to “gesture” typing using the Google keyboard, but that is missing from the AOSP keyboard and no free third party keyboards have it either. I asked the Copperhead guys about it and got this reply:

If the open-source community makes a better keyboard than AOSP Keyboard, we’ll switch to it. Right now it’s still the best option. There’s no choice available with gesture typing, let alone parity with the usability of the built-in keyboard. Copperhead isn’t going to be developing a keyboard. It’s totally out of scope for the project.

So, not a show stopper, but if anyone is looking to make a name for themselves in the AOSP world, a new keyboard would be welcome.

To further increase security, there is a suggestion to create a strong two-factor authentication mechanism. The 6P has a fingerprint sensor, but I don’t use it because I don’t believe that your fingerprint is a good way to secure your device (it is pretty easy to coerce you to unlock your handy if all someone has to do is hold you down and force your finger on to a sensor). However, having a fingerprint and a PIN would be really secure, as the best security is combining something you have (a fingerprint) with something you know (a PIN).

So here was my desktop on OmniROM:

Old Phone Desktop

and here is my current desktop:

New Phone Desktop

Not much different, and while I’ve given up a few things I’ve also discovered OCReader and Nextcloud News, plus the Amaze file manager.

But the biggest thing I’ve gained is peace of mind. I want to point out that it is possible to run other ROMs, such as OmniROM, without Google Services, but they aren’t quite as focused on security as Copperhead. Many thanks to the Copperhead team for doing this, and if you don’t want to go through all the work I did, you can buy a supported device directly from them.

Android Open Source Frustrations

Posted on by Tarus

I used to be a huge fan of Apple products, but as they started to lock down their ecosystem the limitations they created started to bother me, so I switched to running as much open source as possible.

It wasn’t, and isn’t always now, easy. One of the gripes I still have against Apple is that their commercial success has spawned a ton of imitators who have decided to lock down their products, quite often without the Apple savvy to back it up. Unfortunately, Google seems to be joining these ranks.

I’m a fan of Google, they do a lot to support open source, and I use a Nexus 6 as my primary “hand terminal” (handy). However, I run alternative software on it, namely OmniROM, which gives me more control over my experience and security.

I pretty much run open source software on all my technology with few exceptions, one being my Android Wear watch. I noticed that there was a new update to Android Wear (version 2.0) so I went to play with it. When I launched the app I got this screen:

Android Wear App Error

(sigh)

So I went off to search for a solution to the error message “This phone has been flashed with an unsupported configuration for companion. you must re-flash it as either signed/user or unsigned/userdebug”. I found a couple of answers that suggested I edit the build.prop file and change

ro.build.type=userdebug

to

ro.build.type=user

In order to do this, you have to have root access to your phone.

(sigh)

I do root my phone, but I haven’t done it in awhile because Google has introduced this thing called “SafetyNet“. The stated purpose is to prevent malware but in practice what it does is torpedo people like me who actually want to control the software on the devices they own. If you install a custom ROM or have root access, certain applications will not run.

Now I have to choose between running the Android Wear app or, say, Pokémon Go. I chose Android Wear (I pretty much finished Pokémon Go).

The process: Boot into recovery, install SuperSU, boot into system, use a file editor to edit /system/build.prop and change ro.build.type from “userdebug” to “user”, reboot.

Android Wear Mute

So Android Wear will start now, but to add to the frustration the one feature I hoped they would fix is still broken for me. It used to be that if my watch was actively paired with the phone, it would mute ringing and other audio notifications. It doesn’t (and none of the fixes I’ve found work for me) so now I just remember to decrease the volume on the phone down to “vibrate”.

Pokemon Go Blocks root

And, I verified that Pokémon Go will not start now – it hangs on the login screen and then reports an error. This is whether or not SuperSU is enabled, and I think I would have to remove it entirely to get it to work.

Now I know that I can install other apps that will hide the fact that my phone is rooted, but if I do that the terrorists win. I would just rather use apps that don’t force me to give up my rights.

Which brings me to the last frustration. I purchased a bunch of content from Google, but now I can’t access it on this phone. I get “couldn’t fetch license”. This started recently so I believe it has something to do with SafetyNet, but repeated calls to Google Play support yielded no answers.

Google License Error - Deadpool

I have a Google 6P that is stock and doesn’t suffer from the download issue, so it stands to reason that there is some “protection” in place that is preventing me from accessing the content I purchased. I solved the problem by not buying content from Google Play anymore.

I’m pretty certain that it is only going to get worse. Google used to be much better about such things but I think they want to emulate Apple in more ways than one (see the new Pixel phone if you don’t believe me) and that is a shame for all of us.

UPDATE: I found a better way to do this that doesn’t require root. Assuming you have a custom recovery like TWRP, you can simply boot into recovery and then connect the handy to a computer. Using “adb shell” you can then access the system directory and edit the build.prop file directly.

Move to Let’s Encrypt – it’s soooo easy!

Posted on by Tarus

This weekend I wanted to play around with setting up Nextcloud on my home network (we already use it at work and it is awesome). Since I am planning on putting personal information into that app, I wanted to make sure that access to it was encrypted end-to-end.

This meant setting up SSL on my home web server. Now, it used to be that you either had to use a self-signed certificate (which could cause problems) or you had to spend a bunch of money on a certificate from a recognized Certificate Authority (CA).

Enter Let’s Encrypt. Launched in April of this year, Let’s Encrypt provides free certificates that are recognized by most of the things you need to recognize them.

I had been putting it off since dealing with certs is, quite frankly, a pain. You have to fill out a request, send it to the CA, get back a key file, install it in the write place, etc. Even with a free one I didn’t have time for the hassle.

I shouldn’t have worried – with Certbot it is dead simple. Seriously.

Certbot Screen

I went to their site (as directed from the Let’s Encrypt site) and just followed the instructions. I downloaded a script which downloaded all the required dependencies via apt, answered a few questions, and, bam, I had a functioning web server running SSL. They even prompted me if I wanted all requests to port 80 (http) to be redirected to port 443 (https) and when I said “yes” it did it for me.

The whole process took a couple of minutes.

Amazing stuff. The certificates are only good for 90 days, but they even include an automated way to update them.

Certbot Certificate Renewal

As more and more of our personal information becomes digitized, it is extremely important to use strong encryption. In the past this could be inconvenient if not outright difficult, but you really don’t have an excuse with Let’s Encrypt. Use it.

Nextcloud and OpenNMS

Posted on by Tarus

Last weekend, OpenNMS-er extraordinare Ronny Trommer was at a conference where he met Jos Poortvliet from Nextcloud. I’ve been following Nextcloud pretty intently since I recognized kindred souls in their desire to create a business that was successful and still 100% open source (and not, for example, fauxpensource). Jos mentioned that Nextcloud was getting a new monitoring API and thought it would be cool if OpenNMS could use it.

Since their API returns the monitoring information as XML, Ronny used the XML Collector to gather the data. Once the data is in OpenNMS, you can graph it, set thresholds, configure notifications, etc.

Available metrics include:

  • CPU load and memory usage
  • Number of active users over time
  • Number of shares in various categories
  • Storage statistics
  • Server settings like PHP version, database type and size, memory limits and more

Here’s an example of the number of files from a small demo system:

Files in Nextcloud

Of course, since OpenNMS is a platform, once the data is in the system you can leverage its integrations with applications such as Grafana:

Nextcloud Metrics in Grafana

Some applications will go on and on about how many “plugins” they have. Often, these are little more than scripts that do something simple, like an SNMP GET, but with all the overhead of having to run a shell. To add something like Nextcloud to OpenNMS, it is just a simple matter of configuring a couple of files, but to make that easier a lot of configurations have been added to a git repository. If you want to try out the Nextcloud integration, follow these instructions.

True open source solutions can offer the best feature, performance and value for most companies, but unfortunately there are so few pure open source companies providing them. I applaud Nextcloud and look forward to working with them for years to come.

Nagios XI vs. OpenNMS Meridian – the Return of the FUD

Posted on by Tarus

It seems like our friends over at Nagios have been watching a little too much election coverage this year, and they’ve updated their “Nagios vs. OpenNMS” document with even more rhetoric and misinformation.

As my three readers may recall, back in 2011 I tore apart the first version of this document. Now they have decided to update it to target our Meridian™ version.

Let’s see how they did (please look at it and follow along as it is quite amusing).

The first misleading bit is the opening paragraph with the phrase “most widely used open-source monitoring project in the world”. Now, granted, they do indicate that means “Nagios Core” but it seems a little disingenuous since what they are selling is Nagios XI, which is much different.

Nagios XI is not open source. It is published under the “Nagios Open Software License” which is about as proprietary as they get. I’m not even sure why the word “open” was added, except to further mislead people into thinking it is open source. The license contains clauses like “The Software may not be Forked” and “The Software may only be used in conjunction with products, projects, and other software distributed by the Company.” Think about it, you can’t even integrate Nagios XI with, say, a home grown trouble ticketing system without violating the license. Doesn’t sound very “open” at all. OpenNMS Meridian is published under the AGPLv3, or a similar proprietary license should your organization have an issue with the AGPL. You don’t have that choice with Nagios XI.

Next, let’s check out the price. The OpenNMS Group has always published its prices on-line. One instance of Meridian, which includes support in the form of access to our “Connect” community, is $6,000. They have it listed as $25,995, which is the price should you choose the much more intensive “Prime” support option. I’m not sure why they didn’t just choose our most expensive product, Ultra Support with the 24×7 option, to make them seem even better.

Nagios XI Node Limitation

Also, note the fine print “Price based on one instance of XI with 220 nodes/devices”. There is no device limit with OpenNMS Meridian. So let’s be clear, for $6000 you get access to the Meridian software under an open source license versus $5000 to monitor 220 nodes with extreme limitations on your rights.

Our smaller customers tend to have around 2000 devices, which means to manage that with Nagios XI you would need roughly ten instances costing nearly $50,000 (using the math presented in this document). And from the experience we’ve heard with customers coming to us from Nagios, the reason it is limited to so few nodes is that you probably can’t run much more on a single instance of Nagios XI. Compare that to OpenNMS where we have customers with over 100,000 devices in a single instance (and they’ve been running it for years).

We also price OpenNMS as a platform. You get everything: trouble-ticketing integration, graphing, reporting, etc. in one application. It looks like Nagios has decided to nickel and dime you for logs, etc. and a thing called “Nagios Fusion” which you’ll need to manage your growing number of Nagios instances since it won’t natively scale. And remember, due to the license you are forbidden from using the software with your own tools.

I especially had to laugh at the “You Speak, We Listen” part. If you have a feature or change you need, if you ask nicely they might make it for you. With OpenNMS Meridian you are free to make any changes you need since it is 100% open source, and with our open issue tracker we address dozens of user requests each point release.

Finally, there is the feature comparison, which at a minimum is misleading and is often just blatantly false. Almost every feature marked as lacking in Meridian exists, and at a level far beyond what Nagios XI can provide. Seriously, is it really objective to state that OpenNMS doesn’t support Nagvis, a specific tool that even has “Nagios” in the name?

Nagvis

I had to laugh at the hubris. They obviously didn’t Google “opennms nagvis“, because, guess what? There has been an OpenNMS Nagvis integration for some time now, contributed by our community. Just in case you were wondering, we have an integration with Network Weathermap as well.

Nagios is just another proprietary software product that wants to lock you into its ecosystem, and this is just a shameful attempt to monetize an application that is long past its prime. Heck, it was the inability of the Nagios leadership to get along with others that resulted in the very popular Icinga fork, and with it Nagios lost a lot of contribution that helped make up its “Thousands of Free Add-Ons” (and the way Nagios took over the community lead plug-in site was also poorly handled). Plus, many of those add-ons won’t scale in an enterprise environment, which probably lead to the 220 device limit.

Compare that to OpenNMS. We not only want to encourage you integrate with other products, we do a lot of it for you. OpenNMS has great graphing, but we also created the first third party plug-in for Grafana. When it comes to mapping, OpenNMS is on the leading edge, with a focus on various topology views that can ultimately handle millions of devices in a fashion that is actually usable. Need to see a Layer 2 topology? Choose the “enhanced linkd view”. Run VMware and Vcenter? It is simple to import all of your machines and see them in a view that shows hosts, guests and network storage. Plus the unique ability to focus on just those devices of interest allows you to use a map with hundreds of thousands if not millions of nodes.

Nagios Map

Compare that to the Nagios map screenshot where it looks like “localhost” is having some issues. Oh no, not localhost! That’s like, all of my machines.

As for “Business Process Intelligence” I’ve been told that the Nagios XI version is like our Business Service Monitor “Except BSM is more featureful, and has a significantly better UI/UX”. Need real Business Intelligence? OpenNMS has Red Hat Drools support, the open source leader, built right into the product.

We also support integration with popular Trouble Ticketing systems such as Request Tracker, Jira, OTRS and Remedy. And the kicker is that you can also run any Nagios check script natively in OpenNMS using the “System Execute Monitor“, but once you get used to the OpenNMS platform, why would you?

I’m not really sure why Nagios goes out of its way to spread fear, uncertainty and doubt about OpenNMS. We rarely compete in the same markets. I’m sure that Sunrise Community Banks get their money’s worth from Nagios, and for companies like NRS Small Business Solutions, Nagios might be a good fit. But if you have enterprise and carrier-level requirements, there is no way Nagios will work for you in the long term.

When a company does something like this to mislead, from wrong information about our product to using terms like “open” when they mean “closed”, it shows you what they think of their competition. What does it say about what they think about their customers?

Nextcloud, Never Stop Nexting!

Posted on by Tarus

It’s been awhile since I’ve posted a long, navel-gazing rant about the business of open source software. I’ve been trying to focus more on our business than spending time talking about it, but yesterday an announcement was made that brought all of it back to the fore.

TL;DR; Yesterday the Nextcloud project was announced as a fork of the popular ownCloud project. It was founded by many of the core developers of ownCloud. On the same day, the US corporation behind ownCloud shut it doors, citing Nextcloud as the reason. Is this a good thing? Only time will tell, but it represents the (still) ongoing friction between open source software and traditional software business models.

I was looking over my Google+ stream yesterday when I saw a post by Bryan Lunduke announcing a special “secret” broadcast coming at 1pm (10am Pacific). As I am a Lundookie, I made a point to watch it. I missed the start of it but when I joined it turned out to be an interview with the technical team behind a new project called Nextcloud, which was for the most part the same team behind ownCloud.

Nextcloud is a fork, and in the open source world a “fork” is the nuclear option. When a project’s community becomes so divided that they can’t work things out, or they don’t want to work things out for whatever reasons, there is the option to take the code and start a new project. It always represents a failure but sometimes it can’t be helped. The two forks I can think of off hand, Joomla from Mambo and Icinga from Nagios, both resulted in stronger projects and better software, so maybe this will happen here.

In part I blame the VC model for financing software companies for the fork. In the traditional software model, a bunch of money is poured into a company to create software, but once that software is created the cost of reproducing it is near zero, so the business model is to sell licenses to the software to the end users in order to generate revenue in the future. This model breaks when it comes to free and open source software, since once the software is created there is no way to force the end users to pay for it.

That still doesn’t keep companies from trying. This resulted in a trend (which is dying out) called “open core” – the idea that some software is available under an open source license but certain features are kept proprietary. As Brian Prentice at Gartner pointed out, there is little difference between this and just plain old proprietary software. You end up with the same lack of freedom and same vendor lock in.

Those of us who support free software tend to be bothered by this. Few things get me angrier than to be at a conference and have someone go “Oh, this OpenNMS looks nice – how much is the enterprise version?”. We only have the enterprise version and every bit of code we produce is available under an open source license.

Perhaps this happened at ownCloud. When one of the founders was on Bad Voltage awhile back, I had this to say about the interview:

The only thing that wasn’t clear to me was the business model. The founder Frank Karlitschek states that ownCloud is not “open core” (or as we like to call it “fauxpensource“) but I’m not clear on their “enterprise” vs. “community” features. My gut tells me that they are on the side of good.

Frank seemed really to be on the side of freedom, and I could see this being a problem if the rest of the ownCloud team wasn’t so dedicated.

On the interview yesterday I asked if Nextcloud was going to have a proprietary (or “enterprise”) version. As you can imagine I am pretty strongly against that.

The reason I asked was from this article on the new company that stated:

There will be two editions of Nextcloud: the free of cost community edition and the paid enterprise edition. The enterprise edition will have some additional features suited for enterprise customers, but unlike ownCloud, the community and enterprise editions for Nextcloud will borrow features from each other more freely.

Frank wouldn’t commit to making all of Nextcloud open, but he does seem genuinely determined to make as much of it open as possible.

Which leads me to wonder, what’s stopping him?

It’s got to be the money guys, right? Look, nothing says that open source companies can’t make money, it’s just you have to do it differently than you would with proprietary software. I can’t stress this enough – if your “open source” business model involves selling proprietary software you are not an open source company.

This is one of the reasons my blood pressure goes up whenever I visit Silicon Valley. Seriously, when I watch the HBO show to me it isn’t a comedy, it’s a documentary (and the fact that I most closely identify with the character of Erlich doesn’t make me feel all that better about myself).

I want to make things. I want to make things that last. I can remember the first true vacation I took, several years after taking over the OpenNMS project when it had grown it to the point that it didn’t need me all the time. I was so happy that it had reached that point. I want OpenNMS to be around well after I’m gone.

It seems, however, that Silicon Valley is more interested in making money rather than making things. They hunt “unicorns” – startups with more than a $1 billion valuation – and frequently no one can really determine how they arrive at that valuation. They are so consumed with jargon that quite often you can’t even figure out what some of these companies do, and many of them fade in value after the IPO.

I can remember a keynote at OSCON by Martin Mickos about Eucalyptus, and how it was “open source” but of course would have proprietary code because “well, we need to make money”. He is one of those Silicon Valley darlings who just doesn’t get open source, and it’s why we now have OpenStack.

The biggest challenge to making money in open source is educating the consumer that free software doesn’t mean free solution. Free software can be very powerful but it comes with a certain level of complexity, and to get the most out of it you have to invest in it. The companies focused on free and open source software make money by providing products that address this complexity.

Traditionally, this has been service and support. I like to say at OpenNMS we don’t sell software, we sell time. Since we do little marketing, all of our users are self selecting (which makes them incredibly intelligent and usually quite physically beautiful) and most of them have the ability to figure out their own issues. But by working with us we can greatly shorten the time to deploy as well as make them aware of options they may not know exist.

In more recent times, there is also the option to offer open source software as a service. Take WordPress, one of my favorite examples. While I find it incredibly easy to install an instance of WordPress, if you don’t want to or if you find it difficult, you can always pay them to host it for you. Change your mind later? You can export it to an instance you control.

The market is always changing and with it there is opportunity. As OpenNMS is a network monitoring platform and the network keeps getting larger, we are focusing on moving it to OpenStack for ultimate scalability, and then coupled with our Minions we’ll have the ability to handle an “Internet of Things” amount of devices. At each point there are revenue opportunities as we can help our clients get it set up in their private cloud, or help them by letting them outsource some or all of it, such as Newts storage. The beauty is that the end user gets to own their solution and they always have the option of bringing it back in house.

None of these models involves requiring a license purchase as part of the business plan. In fact, I can foresee a time in the near future where purchasing a proprietary software product without fully exploring open source alternatives will be considered a breach of fiduciary responsibility.

And these consumers will be savvy enough to demand pure open source solutions. That is why I think Nextcloud, if they are able to focus their revenue efforts on things such as an appliance, has a better chance of success than a company like ownCloud that relies on revenue from software licensing sales. The fact that most of the creators have left doesn’t help them, either.

The lack of revenue from licenses sales makes most VCs panic, and it looks like that’s exactly what happened with the US division of ownCloud:

Unfortunately, the announcement has consequences for ownCloud, Inc. based in Lexington, MA. Our main lenders in the US have cancelled our credit. Following American law, we are forced to close the doors of ownCloud, Inc. with immediate effect and terminate the contracts of 8 employees. The ownCloud GmbH is not directly affected by this and the growth of the ownCloud Foundation will remain a key priority.

I look forward to the time in the not too distant future when the open core model is seen as quaint as selling software on floppy disks at the local electronics store, and I eagerly await the first release of Nextcloud.

2015 Open Source Monitoring Conference

Posted on by Tarus

Once again I got to visit the wonderful town of Nürnberg, Germany, for the Open Source Monitoring Conference.

OSMC - Badge

Hosted by Netways, the conference started out ten years ago as a Nagios conference. The name was changed due to an issue with the Nagios trademark, but it still focused heavily on Nagios. However, the organizers are pretty open to all things monitoring, so they started inviting projects like Zabbix and OpenNMS to come. When the Nagios fork Icinga was created, the amount of Nagios content dropped considerably, and out of 24 talks over 2 days there were only two that had Nagios in the title. Part of this has to do with Icinga 2 being a total rewrite and thus has started to move past its Nagios roots.

This year it was a cornucopia of monitoring choices. In addition to Icinga, Zabbix and OpenNMS, there was Alyvix, Assimilation, Heroic, and Prometheus. Grafana was popular and most tools are adding support for that data visualization tool, and it was nice to see talks on NSClient++ and MQTT. A little less than half the talks were in German, so there is a large German focus to the conference, but there was always an English-language talk available as well.

Nürnberg is a cool town. There is a big castle and lots of walls are left over from the original fortifications for the city. It is also home to SuSE Linux, and I made sure to swing by if just to get a picture for Bryan Lunduke:

OSMC - SuSE Office

Ronny and I got there on Monday. While the main conference is held over two days, this year there were workshops on Monday and a “hack-a-thon” on Thursday. The conference pretty much takes over the Holiday Inn, City Center, hotel. While the facilities are nice, it is right next to the city’s “eros center” which seems to creep closer and closer to the hotel each year I attend. It doesn’t impact the conference in any way, and those who might be sensitive to such things can easily avoid it.

There is always lavish catering and this year we had a nice, small crowd of OpenNMS enthusiasts in attendance, and we met up for the hosted dinner on Monday night. I had not seen some of the people since the OUCE, so it was nice to catch up.

My talk was on Tuesday, the first day of the main conference. The event was sold out, with about 250 people, and at times the rooms could get quite full.

OSMC - Crowd

The talks were all rather good. Torkel Ödegaard talked about Grafana:

OSMC - Grafana

which was a big hit with crowd, and as I mentioned before a lot of projects are leveraging his work to provide better data visualization, including OpenNMS. My talk went well (I think) as I went over all of the amazing things we’ve done since last year at the OSMC, which included four major releases of our application. I was stumped with the question “How do I get started with OpenNMS?” when I realized that I didn’t have an easy answer. I can tell you how to install it, but that doesn’t get you started. I need to work on that.

That evening we returned to Terminal 90, which is an odd place to hold a dinner but it seems to work. Terminal 90 is a restaurant located at the Nürmberg airport, and it does a good job of holding everyone. We have to take the U-bahn to get there, and at least this year there were no incidents (last year someone tried to hold open the doors, which caused the autonomous train to shut down and wait for human intervention).

OSMC - Terminal 90

The food and drinks were good, and toward the end of the evening they had woman impersonating German pop star Helene Fischer, which was lost on me but the crowd seemed to enjoy it.

I called it a night fairly early, but this is a group that tends to hang out until the wee hours of the morning. Although my room was on the first floor, I didn’t hear much noise from “Checkpoint Jenny” across the street, so maybe everyone is getting more mellow in their old age. (grin)

The second day featured a number of talks from different projects. Usually the Zabbix talk is done by Rihards Olups, but he was unable to make it this year so Wolfgang Alper did the honors.

OSMC - Zabbix

After that was a really good talk by Martin Parm on how Spotify monitors its music service.

OSMC - Spotify

It started out with all of the tools they tried that failed, and I kept thinking to myself “don’t let it be OpenNMS, don’t let it be OpenNMS” (it wasn’t) and ended with a tool they wrote in-house called Heroic. It is a time-series data store built on top of Cassandra, and it looks a lot like the Newts tool we built. Both are open source and Apache-licensed so I’m hoping to find some synergy between the two projects. There is another large music streaming service that uses OpenNMS, but maybe we can get all of them (grin).

OSMC - Prometheus

Then there was a talk by Fabian Reinhartz on a monitoring system called Prometheus. I had to joke that the name refers to the daily experience of most network managers of having their liver eaten out, but it seems like an interesting tool. Written in Go, it may find resistance from users due to the configuration being more like writing code, but that also makes it powerful. Sounds familiar to me.

I had to leave right after lunch in order to be ready to catch my flight home, but I really enjoyed my time there, even more than usual. Many thanks to Bernd Erk and the Netways gang for holding it, and they should be posting the videos soon. If you are interested in next year be sure to register early as it is likely to sell out again.

Open Source Software and Corporations

Posted on by Tarus

An interesting post caught my eye this week entitled “Corporations and OSS Do Not Mix” by Ian Cordasco. It was kind of depressing – here was a person who had spent a lot of free time contributing to open source code, but the actions of some users of that code had taken the fun out of it.

My only issue with it was the targeting of “corporations” in the title. At OpenNMS we have a large number of corporate customers and we get along with them just fine. I want to talk about that in a bit, but first I want to address some of the other experiences Ian had that were similar to mine.

When I became the maintainer of OpenNMS back in 2002, I would often get e-mails from people that would start out with “OpenNMS is good, but what you need to do is …”. I used to spend a lot of time responding to them, pointing out that it was open source and anyone can help contribute to it, so they didn’t have to wait on me to do anything, but it never really helped and it turned into a huge time suck. I started to send back a generic e-mail that went along the lines of “OpenNMS is an enterprise product and if you won’t take the time to understand it then you should try something easier like Nagios” which would usually result in a reply calling me an asshole, but it took little of my time and then conversation was over. Now I pretty much just ignore them.

When you create something and share it, you are putting a bit of yourself out there and there are bound to be critics. For the most part they can be ignored, and you have to develop a thick skin to be in this environment. I’ve found that overall the good far outweighs the bad, and if you can learn to brush off the bad you can be very happy working in open source.

People tend to forget that open source “business” is still “business”. People exchange money in return for services. If I had Ian’s talent I would simply set up various custom development options, so when someone complained about a bug he could just return an e-mail with a price list. If you don’t have time to do it, make the prices really, really large – large enough that you would make time to do it. It’s your life – you are in the driver’s seat. I used to give a talk on running an open source business and I always stressed that you should never compete on price, or at least you shouldn’t lead with “my solution is cheaper”. Sure, open source software can provide tremendous savings over the life of the solution, but that doesn’t mean the solution itself is inexpensive to get set up. Done right, it will be better than any proprietary solution, but that doesn’t mean it comes without cost.

Always remember: free software does not mean free solution.

Getting back to dealing with corporations, like any interaction between two parties is it extremely important to set up expectations. You need to clearly outline what the product the client is buying covers (response time, 24/7 support, etc.). If they aren’t buying anything, then you don’t need to worry about them. I chuckled when I read “Well if you’re not going to take this seriously, we’ll have to start using another project.” We often get the “use another project” line and my response is “knock yourself out”. If you want to take this seriously, then pay me for my work. It’s like going into a free kitchen and complaining the soup is too salty.

A more difficult issue comes when someone wants to submit substandard code. This does require a little effort, since you can’t be sure that this isn’t just an eager but inexperienced coder versus someone lazy. Again, expectations are important. If you publish what the base level of quality should be, such as “must include unit tests”, then you can point to that when you don’t accept a submission. Plus, git makes it very easy to track a master branch and just apply your changes, so some sort of reply about how to do that could deflect criticism about the speed in accepting pull requests.

Ian makes a lot of really good points in his post, but I think he misses a point that if you run your open source project like a business then corporations (i.e. other businesses) will respect you and treat you like a business. We have one amazing company that just hired four (!) OpenNMS developers to work on code that they need. While some of it, if not most of it, will address their particular needs, all of it will be put into OpenNMS and they are paying us (gasp) to help project manage that team. That relationship did not happen overnight, but was built on a series of successful projects where we delivered particular value in exchange for money.

Look, I love, by and large, the open source community and I like being a part of it, but that doesn’t mean that open source and business are mutually exclusive. Learning to deal with open source as a business not only insures more open source gets created, but it also keeps it fun.

♫ The Lunatic is on My Web ♫

Posted on by Tarus

The TL;DR of it is that I needed to create a new forum called OpenNMS Connect. This will be a place for Luna. So far I’ve been happy.

When I first started my quest for forum software a couple of month ago, I did what most geeks do and did a search for it. I found a very helpful Wikipedia page (‘natch).

After dismissing the non-open source options, I started looking at the programming language. Now I know I really shouldn’t be a PHP snob (this blog is presented using PHP software) but having been burned in the past with security issues my first inclination is to avoid it.

Now the guys in the office are trying to get me to think all “agile-ly” and so I need a “user story”. For any forum we use it has to support LDAP, for which the story could be “User must be able to access forum using directory services” or better yet “Admin needs a central way of controlling forum access”. We implement LDAP via the FreeIPA project, and it will just be so much easier if we can add and remove people from a particular group and just have it work.

The first project I looked at was Discourse. I was especially interested in a hosted version if I could tie it into our IPA instance. Discourse is kind of the “new hotness” at the moment, but I didn’t see an easy way to implement LDAP. There is a Single Sign On (SSO) option but it would require writing our own authentication page, and it wouldn’t work if we hosted it with them anyway.

The next project that caught my eye was the eXo Platform. It’s written in Java (as is OpenNMS) and it seems to have a ton of features. Perhaps too many. In any case I put the team on it and asked them to get it working with LDAP.

They succeeded in getting LDAP authentication to work, but then hit a ton of other snags. The authenticated users couldn’t access the default /portal/intranet site no matter how often we tweaked the permissions. They could reach the /portal/meridian site but we couldn’t figure out how to change the default portal. And in all cases we couldn’t get the top menu bar to load with an LDAP user which meant you couldn’t log out, etc.

On Friday I decided to see what I could do about it. Friday was a long day.

eXo is one of those companies that produces an open source version of their software as well as a paid version. My three readers know how I feel about that business model, and it made it kind of frustrating to figure out things since I couldn’t tell if the documentation would actually work on the “community” version. Also, to access the forums you need to register, which gets you a couple of spam-y e-mails trying to sell you on their paid version. Not too obnoxious and I can understand why they do it, but it was a little annoying.

It can also be hard to administer. A lot of the configuration is buried in .war files. For example, in order to set the default portal above, you have to unpack portal.war, change it and repack it. In playing around with the system, I decided that while the LDAP authentication is nice, the platform itself is way overkill for what we need. It is huge and on our system took several minutes to start up and would often spike the load with limited users.

So I spent a lot of time looking for alternatives. Unfortunately, the only option I found that had easy to understand LDAP integration was phpBB. When I mentioned that to the team, Jeff threw up in his mouth a little and I wasn’t too happy about that choice either. I don’t have the same prejudices as some, but I felt that its style was a little dated and there have been some serious security issues in the past associated with it.

But for grins I installed phpBB anyway. It was rather easy to do, which made me happy, but then I noticed that it was not easy to make the forum itself private. Another user story is that “Admin requires that only authorized users see the forum”. You can make certain parts of phpBB private, but I kind of wanted the same thing as eXo – an initial log in screen you have to use before accessing the site.

Then it dawned on me that we could just put it in a directory by itself in the web root, say /forum, and then make a pretty splash page on on the site with a link to it. Apache LDAP authentication is something we already figured out and knew worked and I could just require a valid login to access /forum.

This caused another lightbulb to go off. If we are going to do it that way, then why not just put any forum we like behind an LDAP authenticated directory?

The downside would be that users would need to create a forum-specific user if they wanted to add content, but on the upside they could choose their own usernames, thus obfuscating their identities for people who work at sensitive organizations. Thus we could have an LDAP user tied to, say, obama@whitehouse.gov and their forum name could be something totally different, like “Hot Cocoa”.

Yes, I know it is dressing up a bug as a feature, but to me it did seem useful.

Then I thought, hey, let’s revisit Discourse. That turned out to be harder than it would seem

Well, the only way to install Discourse on CentOS is as a Docker container, and at the moment it doesn’t seem to work.

The first time I tried to install it, it died complaining about lack of access to an SMTP server. No where in the instructions did it say you had to modify the app.yml and put in a valid mail server. In any case, I did that and restarted the install.

At one point during the install process I get this:

-- 0:  unicorn (4.8.3) from
/var/www/discourse/vendor/bundle/ruby/2.0.0/specifications/unicorn-4.8.3.gemspec
Bundle complete! 92 Gemfile dependencies, 189 gems now installed.
Gems in the group development were not installed.
Bundled gems are installed into ./vendor/bundle.

I, [2015-04-04T04:49:47.161747 #38]  INFO -- : > cd /var/www/discourse
&& su discourse -c 'bundle exec rake db:migrate'
2015-04-04 04:49:55 UTC [339-1] discourse@discourse ERROR:  relation "users" does not exist at character 323
2015-04-04 04:49:55 UTC [339-2] discourse@discourse STATEMENT:      SELECT a.attname, format_type(a.atttypid, a.atttypmod),	                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
	                FROM pg_attribute a LEFT JOIN pg_attrdef d
	                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
	               WHERE a.attrelid = '"users"'::regclass
	                 AND a.attnum > 0 AND NOT a.attisdropped
	               ORDER BY a.attnum

which a Google search says to ignore, but then a little while later the install fails with:

FAILED
--------------------
RuntimeError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #
Location of failure: /pups/lib/pups/exec_command.rb:105:in `spawn' exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
68a9a49f29ad74d9ab042bcaadfb06e02ff526104fefd82039eae1588bbb6e43
FAILED TO BOOTSTRAP

on which Google is much less helpful. No matter what I did I couldn’t get past it.

This kind of brings up an issue I have with Docker. Now let’s get this out of the way: I am jealous of the Docker project. We’ve been around for 15 years and gotten little notice whereas they have become huge in a short time. It would be nice if, say, I could get up to four readers on my blog.

But I really, really, really hated how hidden this whole process was. You install software on your system and then load “magic bits” from the Internet and hope it works. I think this is great on a intranet when you need to deploy lots of the same things, but without developing it internally first it was a little scary. When it doesn’t work it is incredibly hard to diagnose. Because the app wouldn’t build I couldn’t play with the database or really do anything, so I just uninstalled and reinstalled numerous times to try to fix this.

Plus, by running in a container, we would then need to modify nginx to use our LDAP configuration and that seems to be much harder than with Apache. I didn’t think it would be easy to just forward requests to the Docker instance, but since I couldn’t get it to work I’ll never know.

By this time I said, screw it, reinstalled phpBB and went home. It’s now about 8pm and I’ve been at it 11 hours.

Well, I have a mild form of OCD, or maybe it’s just being a geek, but I couldn’t let it rest. So early this morning (as in soon after midnight) I discovered a project called Luna (an active project from the aforementioned Wikimedia page).

Luna is the next iteration of the ModernBB project which is in turn is a fork of FluxBB. It’s simple, does almost everything I could want, and was incredibly easy to install. No Docker containers, no large Java app, just some PHP that you drop in your web root. Plus the webUI is built on bootstrap just like OpenNMS.

In about an hour I had it running, had changed the style to match our color palette, and fixed an issue where jquery wasn’t getting loaded by copying it down as a local file.

OpenNMS Luna Website

The downside is that it isn’t production yet. I installed 0.7 and earlier this morning they released 0.8. Jesse fixed an issue with the internal mail system and I have a couple of more issues that I’d like to see fixed, but overall I’m very happy with it. They are aiming to release 1.0 on 13 April.

And I really like their attitude and philosophy. They are self-funded and I love Yannick’s tag line of “You Can Do Anything.”

To help that I sent them 100€. (grin)

Anyway, sorry for the long post. I’ll let you know how it goes.