Month: November 2015

2015 Open Source Monitoring Conference

Posted on by Tarus

Once again I got to visit the wonderful town of Nürnberg, Germany, for the Open Source Monitoring Conference.

OSMC - Badge

Hosted by Netways, the conference started out ten years ago as a Nagios conference. The name was changed due to an issue with the Nagios trademark, but it still focused heavily on Nagios. However, the organizers are pretty open to all things monitoring, so they started inviting projects like Zabbix and OpenNMS to come. When the Nagios fork Icinga was created, the amount of Nagios content dropped considerably, and out of 24 talks over 2 days there were only two that had Nagios in the title. Part of this has to do with Icinga 2 being a total rewrite and thus has started to move past its Nagios roots.

This year it was a cornucopia of monitoring choices. In addition to Icinga, Zabbix and OpenNMS, there was Alyvix, Assimilation, Heroic, and Prometheus. Grafana was popular and most tools are adding support for that data visualization tool, and it was nice to see talks on NSClient++ and MQTT. A little less than half the talks were in German, so there is a large German focus to the conference, but there was always an English-language talk available as well.

Nürnberg is a cool town. There is a big castle and lots of walls are left over from the original fortifications for the city. It is also home to SuSE Linux, and I made sure to swing by if just to get a picture for Bryan Lunduke:

OSMC - SuSE Office

Ronny and I got there on Monday. While the main conference is held over two days, this year there were workshops on Monday and a “hack-a-thon” on Thursday. The conference pretty much takes over the Holiday Inn, City Center, hotel. While the facilities are nice, it is right next to the city’s “eros center” which seems to creep closer and closer to the hotel each year I attend. It doesn’t impact the conference in any way, and those who might be sensitive to such things can easily avoid it.

There is always lavish catering and this year we had a nice, small crowd of OpenNMS enthusiasts in attendance, and we met up for the hosted dinner on Monday night. I had not seen some of the people since the OUCE, so it was nice to catch up.

My talk was on Tuesday, the first day of the main conference. The event was sold out, with about 250 people, and at times the rooms could get quite full.

OSMC - Crowd

The talks were all rather good. Torkel Ödegaard talked about Grafana:

OSMC - Grafana

which was a big hit with crowd, and as I mentioned before a lot of projects are leveraging his work to provide better data visualization, including OpenNMS. My talk went well (I think) as I went over all of the amazing things we’ve done since last year at the OSMC, which included four major releases of our application. I was stumped with the question “How do I get started with OpenNMS?” when I realized that I didn’t have an easy answer. I can tell you how to install it, but that doesn’t get you started. I need to work on that.

That evening we returned to Terminal 90, which is an odd place to hold a dinner but it seems to work. Terminal 90 is a restaurant located at the Nürmberg airport, and it does a good job of holding everyone. We have to take the U-bahn to get there, and at least this year there were no incidents (last year someone tried to hold open the doors, which caused the autonomous train to shut down and wait for human intervention).

OSMC - Terminal 90

The food and drinks were good, and toward the end of the evening they had woman impersonating German pop star Helene Fischer, which was lost on me but the crowd seemed to enjoy it.

I called it a night fairly early, but this is a group that tends to hang out until the wee hours of the morning. Although my room was on the first floor, I didn’t hear much noise from “Checkpoint Jenny” across the street, so maybe everyone is getting more mellow in their old age. (grin)

The second day featured a number of talks from different projects. Usually the Zabbix talk is done by Rihards Olups, but he was unable to make it this year so Wolfgang Alper did the honors.

OSMC - Zabbix

After that was a really good talk by Martin Parm on how Spotify monitors its music service.

OSMC - Spotify

It started out with all of the tools they tried that failed, and I kept thinking to myself “don’t let it be OpenNMS, don’t let it be OpenNMS” (it wasn’t) and ended with a tool they wrote in-house called Heroic. It is a time-series data store built on top of Cassandra, and it looks a lot like the Newts tool we built. Both are open source and Apache-licensed so I’m hoping to find some synergy between the two projects. There is another large music streaming service that uses OpenNMS, but maybe we can get all of them (grin).

OSMC - Prometheus

Then there was a talk by Fabian Reinhartz on a monitoring system called Prometheus. I had to joke that the name refers to the daily experience of most network managers of having their liver eaten out, but it seems like an interesting tool. Written in Go, it may find resistance from users due to the configuration being more like writing code, but that also makes it powerful. Sounds familiar to me.

I had to leave right after lunch in order to be ready to catch my flight home, but I really enjoyed my time there, even more than usual. Many thanks to Bernd Erk and the Netways gang for holding it, and they should be posting the videos soon. If you are interested in next year be sure to register early as it is likely to sell out again.

The Inverter: Episode 54 – The Trolley Problem

Posted on by Tarus

Throw out the first segment, and this is one of the best Bad Voltage episodes yet.

It’s not that the first segment sucks (well, for certain values of “suck”), but it pales in comparison to the rest of the show.

That first bit concerns a rant, introduced by Aq, about a trend in programming to rely on “frameworks” instead of actually learning how to code in a particular language. It was set off, as I understand it, by someone wanting to know how to add together two numbers using JQuery, and the response was, uh, why don’t you just add the numbers together using Javascript?

I can understand the frustration. There was a recent rant by Linus Torvalds about a pull request submitted against the kernel that was unnecessarily obtuse. As the pressure mounts to get more and more code out faster and faster, not only are novice programmers being asked to do more complex tasks, they are relying more and more on frameworks and libraries to do them.

While I am not a coder, I do view the writing of code as an art form, and I like code that is artistic: beautiful, clever and functional. I can remember many years ago visiting an especially ugly page on a government website, and when I looked at the source I found it had been generated by Microsoft Frontpage. Yes, that tool would create a web page, but in no way will the code be beautiful or clever, or in this case, functional.

I was not sure if this rant applied to IDEs. Almost all OpenNMS code is done in Eclipse. I think I’m the only one who uses vi, along with healthy amounts of recursive grep. We also use a lot of libraries. Why reinvent the wheel? Of course, this has caused the size of the OpenNMS application to balloon, currently pushing more than half a gigabyte. But space is relatively cheap and time matters, so why not?

I thought it very telling when Aq decided he disliked code that involved any level of abstraction above what he was using. It reminded me of the old George Carlin joke that anyone who drives faster than you is a maniac, while anyone who drives slower than you is an idiot. I did like it when they reminisced about classic code that was very compact and just plain fast. These days we trade speed of completion for speed of execution. My own memory is of running Mac OS 6 on one double sided (800K) floppy. I could put the O/S, MacPaint, MacDraw and MacWrite all on one disk will about 100K left for my files. I couldn’t afford a Mac back then (they ran about US$5K) but the school had ones you could use and all I needed to carry was that disk.

The next segment talked about the Blue Yeti microphone. I bought one of these specifically for the time I was on Bad Voltage, so there must be something in the water about this show and owning one. I was a little confused, however, when the segment starts and Jono states he bought his as a travel mic. This sucker is huge, and as I like to travel as light as possible I can’t imagine dragging it around. However, as the segment continues, it is obvious we are talking about the same mic.

It is a great device. While I like getting input from the gang on which toys to buy, my go-to source for tech advice is The Wirecutter, and the Yeti is their microphone of choice as well. If you plan on recording for the Internet, you should seriously consider getting one of these.

It is the third segment that I thought was brilliant. I’m not sure who came up with the idea, but the discussion centered around ethics programming in self-driving cars. While I disagree with Jeremy that this is something that will need to be figured out before these vehicles become mainstream, it will be a question in need of an answer as they mature.

The scenario offered is this: You are in your self-driving car going along a mountain road. Suddenly, you turn a corner and there are five people in the way. Assuming the car can detect this, should it continue on, protecting the passenger but possibly killing the five people, or should it drive over the side of the cliff, killing the passenger but saving the people in the road?

Wow – what a neat question.

I have no idea of the correct answer. It did dawn on me (as it did the gang) that if the solution was to sacrifice the passenger that pranksters would be more than happy to jump in front of these cars just to see what happens, and I think in at least those models aimed at higher end consumers, they may tout that passenger safety has been programmed into the system to be paramount.

It was a real “grown up” question and I think spawned one of the better discussions ever done on the show.

I was surprised no one brought up Spock’s death speech, “The needs of the many, outweigh … (the needs of the few) … or the one” but Aq did reference the I, Robot movie so he gets points for that.

The final segment concerned the UK government’s decision to put pressure on technology providers to eschew strong encryption in favor of either weak encryption or some sort of back door. Apple has stood up and stated that, if enforced, they would stop selling their products in the UK. It was scary to think about this, since no elected official in any company would want to be labeled as the guy who stood in the way of someone getting an iPhone. Bryan pointed out that the market capitalization of Apple is roughly US$700B, putting it at about 25% of the UK’s GDP (with its fifth highest GDP in the world), and so that threat carries a lot of weight.

This was another “big boy question” and I liked the discussion. Should anyone announce that a back door exists in a popular technology, you can bet the bad guys will throw everything at exploiting it. It’s just not a good idea, although it isn’t surprising that it comes from the UK, a country known for the ubiquitous use of CCTV (on a side note, they have also started using traffic cameras that track you between points and if you exceed the posted speed between them, you get ticketed.)

Of course, there is the thought that a private company like Apple has the ability to sway governments, but no one minds the 800 pound gorilla when it is on your side.

During the outro the guys announced they are returning to SCaLE next year to do a Live Voltage show. These are awesome and shouldn’t be missed, and they have room for nearly 1000 people in the venue so expect it to be crazy. Plus, if you visit the site you’ll see Bryan Lunduke right on the front page next to Cory Doctorow – which I think is pretty cool. Outside of Live Voltage, he’ll be doing a presentation on why he hates freedom, I mean, why Linux sucks.

While we aren’t sponsoring that show, OpenNMS is a gold sponsor at the conference, so be sure to go and stop by our booth.

Anyway, the lads did a great job this week. If you have never listened to Bad Voltage, this would be a great one with which to start.

Review: Signal by Open Whisper Systems

Posted on by Tarus

I like security, and one of the biggest security holes in my technology concerns text messaging and phone calls. While I can secure my data (for the most part), it is hard to secure traffic over the telephone network, especially with the proliferation of devices like the Stingray.

Awhile ago my friend Jeff introduced me to Red Phone by Open Whisper Systems, which was an app that would encrypt your phone calls. I could never get it to work very well, so I didn’t use it, plus Jeff was the only person I talked with who used it.

Flash forward more than a year, and I’m finding that I quite often don’t get texts from Jeff, while he gets mine just fine. He did some investigation and traced the issue to TextSecure, which was an encrypted text app also from Open Whisper Systems. Apparently I was registered on his phone as a TextSecure user, so it was trying to send text to me by that method. Since I no longer had Red Phone on my device (I play a lot with the software on my mobile devices and had not restored it after a clean install) I wasn’t getting the messages.

I went to install TextSecure and found that it has been replaced by Signal. My, what a difference a year makes. Not only was it easy to use, the app itself is pretty nice. It combines both TextSecure and Red Phone features, and is now the default SMS application on my handy.

Signal is 100% open source. The only way for true security is if everyone has the opportunity to examine the code and look for vulnerabilities. Plus, think about it, if you care about security chances are you want to send sensitive information using the service. Without open source you can’t be sure that information isn’t being intercepted by third parties.

This has resulted in some pretty high endorsements:

Quotes about Signal

Signal is available for both Android and iOS, Note that is uses a data connection to send encrypted SMS messages, so it will count against your data cap. I haven’t had the chance to try out the phone functionality as of yet, but it works fine as a normal SMS client as well.

It is nice to come across such a useful piece of software that is 100% open source, and if I happen to send you SMS messages, be on notice that I will be sending you an invite to Signal (grin).

UPDATE: This is so cool. Since the app uses data instead of the SMS protocol for encrypted texts, it works as long as the mobile device has data. Which means that I can get texts no matter what SIM card is currently in my handy. Cool! So I’m in Germany using my Ortel SIM and I’m able to get SMS messages from friends in the US who have no idea where I am or what network I’m using. Killer feature.

Reflections on Paris and My Cowardice

Posted on by Tarus

I was on a bus in Ireland when I heard the news about the Paris attacks. I had gotten up early to head to the opposite coast as I wanted to see an Ireland that wasn’t Dublin, and I don’t think I could have picked a better spot than Doolin, in County Clare.

Today was to be a particularly gray day and it was dark when I started out. It didn’t get much lighter as we rode to Galway, and when I changed buses the driver was playing the news from the radio. Of course the only story was about the more than one hundred people killed in senseless violence overnight.

Peace Symbol by @jean_jullien

I have some friends in Paris and so I immediately reached out to them. As I waited for a response, I pretty much sat, stunned, as the Irish countryside passed by outside my window.

Once I got to my B&B, I dropped my bag and took a long walk, looking for lunch. The day reflected my mood perfectly. It was like nature itself was in mourning. At high noon the sky wasn’t much lighter than at dusk. A roaring wind came off the sea, churning up angry whitecaps. The clouds drizzled rain like tears.

By the time I was getting cold, I found the recommended pub and went in. It was packed, as this is a popular tourist location and they drop people off by the bus load. Since I was alone, I offered to sit at the bar to make room for the next coach, which arrived about five minutes after I did.

A boisterous crowd of mainly young people came in and crowded around the bar where I sat. They were laughing and joking, blissfully unaware of how quickly that can change. I took a little comfort in the normalcy of that moment: people ordered food, the Indian guy asked about vegetarian options, and drinks were poured (including an inexplicable request for a bottle of Miller beer).

As I ate my meal, a nice smoked salmon salad and a wonderful seafood chowder stuffed with mussels, I was reminded of the last time I had mussels this good, which just happened to be in a Belgian restaurant in Paris called La Gueuze.

And I struggled with a dilemma. The Paris Open Source Summit is next week and I am supposed to be there. Heck, I lobbied hard for the opportunity to participate. But while the chance of anything happening is very slim, I can’t say I’m eager to be in Paris at the moment, especially as part of a large crowd.

So I decided not to go.

There were a number of factors. Part of it was concern for my wellbeing. Part of it was concern for my family. I travel a lot and I know they worry no matter where I’m going, and they have been very understanding when I’ve gone to places that don’t exactly have a reputation for safety. I refuse to put my decision on them, but it did play a role.

But I think the deciding factor was actually how much I enjoyed Paris on my last trip. It is an amazing city, and I didn’t want that memory ruined by seeing soldiers on every corner or having to go through intrusive screening at every point of entry.

It makes me feel like a coward. The terrorists have won.

And I can’t understand it. Of all the countries in Europe, the French bend over backwards to be accommodating to different views and ways of thinking. The French motto “Liberté, égalité, fraternité” leads with the word for freedom, and they go to great lengths to explore all the weird corner cases to insure their society is as free as possible.

And that’s what makes me the most angry. I’m certain these acts are going to change that. Not only will it move France to be more restrictive, it will give the more aggressive countries reason to step up military action in the Middle East. A lot more people will die, and most of them will have darker skin. This will create more terrorists, and the cycle will continue.

I hope France and the rest of the world shows some restraint. I’m not, in any way, shape or form, suggesting justice not be sought out, but I’m reminded of something I saw many years ago.

I was living at my parents’ house and my two-year-old nephew was staying with us. It was a beautiful day and so the windows were open, and there was a gentle breeze throughout the house. One strong breeze caught the door behind the boy and slammed it shut. It scared him, so he reached out and smacked the door, as if to punish it. It struck me as a perfect example of a childish reaction – I’m scared and angry so I need to strike out at the nearest thing, whether is makes sense or not.

I hope the world remembers that we are not children.

I don’t have any answers on how to make things better. The best I can do is to promote free and open source software. I know it sounds silly, using FOSS to cure the world’s problems, but in every place I’ve visited (and I’ve been to 37 different countries) I’ve found like-minded people in that community with a strong desire to create new things through cooperation. It creates an environment where anything is possible. In a small way, it creates hope.

I am writing this sitting on my bed at the B&B. It’s cold, and the wind is whipping around the house, but I feel cozy and safe. Here’s a wish that everyone can find a place to be cozy and safe, as well as the hope that tomorrow will be a better day.

Horizon 16.0.4 Security Release

Posted on by Tarus

In response to the Apache Commons library that OpenNMS uses, version 16.0.4 has been released to help secure against a remote exploit.

The exploit involves Java Remote Method Invocation (RMI) which listens on port 1099 by default. In my previous post I pointed out that if that port is inaccessible, then the exploit can’t happen.

What 16.0.4 does is limit RMI to only listen on localhost. While that will prevent remote exploits even in the event port 1099 is blocked via the firewall, it doesn’t completely solve the problem. To fix the root cause of the issue will require changes to Apache Commons, and we are ready to upgrade to the fixed version as soon as it is available.

We tend to be very internally critical of security issues within OpenNMS, and some people complained that my last post wasn’t technical enough. So I’m hoping to correct that with this one, but if you don’t care about such things you should probably skip it (grin). I have started updating the Security Considerations page on the wiki with details about securing OpenNMS in general, and that will have better information for people interested in security and OpenNMS than this blog post.

While blocking external access to port 1099 will secure OpenNMS against this attack for most people, it doesn’t prevent people who have access to the machine from exploiting the vulnerability. This is called a “privilege escalation” attack vs. a “remote exploit”, as a “normal” user can now have rights (i.e. root access) if they are locally on the machine. Most of our users tend to limit shell access to the server, so this shouldn’t be a problem, but in environments that rely heavily on directory services such as LDAP, the default may be to allow non-privileged access to certain users (say, the “IT Group”) that aren’t involved in maintaining OpenNMS.

And there is also the slim chance that there is a vulnerability in our webUI that could allow a user access to the system. We, of course, don’t know of any and we take great care to prevent it, but simply hoping to limit access to the server as a way to prevent this exploit is insufficient.

So, to prevent it entirely, we are removing RMI. It was introduced in the first iteration of the OpenNMS Remote Poller, but real world installation found that getting the proper ports open was a real pain. So instead the remote poller now talks over HTTP/HTTPS (with the latter being the most secure). Most networks have ports 80 and 443 open, so that made things a lot easier.

Until that is introduced (most likely with Horizon 17), it is still a good idea to limit access to the OpenNMS server to only essential people.

Note that Java Management Extensions (JMX) also use serialized objects and thus could be vulnerable. OpenNMS has a JMX port (18980) but it is bound to localhost by default. In fact, all ports are bound to localhost by default in 16.0.4 except for the webUI, port 8980.

There are a number of other steps you can take to harden your OpenNMS server. I’m planning on detailing them on the wiki, but start with only doing a minimal operating system install. The less software on the system, the smaller the chance one will have a vulnerability.

Also, OpenNMS currently runs as the “root” user. This is due to the fact that it needs access to ICMP traffic as well as port 162 for SNMP traps. Both of these require root by default. With some “stupid kernel tricks” you can run OpenNMS as a non-root user, but it has not been heavily tested. We have a detailed list of issues for running as non-root on our Jira instance.

Sorry to drone on about this, but we take security extremely seriously at OpenNMS. We also have to labor under the misconception that Java is inherently unsafe. It is not true, although people still have nightmares from the early issues with client-side Java applets. The Java in OpenNMS is server-side and we don’t use applets, and the language is used securely in a tremendous amount of software.

For comparison, WordPress, an application I love, is currently estimated to run 25% of the world’s websites. It is written in PHP, a language that has a huge track record of security exploits, and many of the spam e-mails I get link to compromised WordPress sites.

It is possible to secure WordPress (we use it for all of our websites as well) but it takes some diligence. We will remain as diligent as we can concerning the security of OpenNMS, and we will continue to take steps to make it even more secure.

Dublin OpenNMS Meetup

Posted on by Tarus

I’m working in Ireland this week, and our UK/Irish Ambassador, Dr. Craig Gallen, used the opportunity to put together an OpenNMS meetup, featuring beer and pizza (grin).

We held it in an office space near Temple Bar thanks to Barry Alistair. Among his many talents, he is also one of the organizers behind IrishDev.com, an on-line community for the Irish Software Developers Network.

Ulf at Dublin Meetup

It was a lot of fun. We socialized for a bit, and Craig had arranged the pizza to arrive at the end of our talks in order to reward folks for listening to us hold forth on the wonders of OpenNMS (the beer was on offer first, ‘natch). Once again I ran long and the pizza was consumed between my introduction and Craig’s presentation. I did an overview of the history of OpenNMS and why using open source, especially for a network management platform, is a Good Thing™.

Craig at Dublin Meetup

Craig’s presentation was much better, and covered a lot of the new features that have recently been added to the application as well as the direction the product was moving (such as being positioned for SDN/NFV/Internet of Thingies). No one left or fell asleep and there were lots of good questions.

Events such as this are one of my favorite things to do, so I want to thank Barry and Craig for making it possible.

The Many Uses of Grafana

Posted on by Tarus

One of the things I love about open source and OpenNMS in particular is watching what people do with it. We knew that we had a great data collector in OpenNMS but sometimes it was hard to display that data in a useful fashion.

OpenNMS is a platform and it is very broad. For example, we do log management, but that is only a small portion of what the application can do, yet there are companies who do nothing but that. So yes, we can display graphs but we don’t necessarily have the resources to focus on making a great data visualization tool.

Enter open source. Torkel Ödegaard has written a great visualization tool in Grafana, so it would be silly for us not to leverage it.

I was at a customer site I and I saw this cool graph:

Grafana Graph

I asked Patrick about it, and he said that he wanted to play with the OpenNMS/Grafana integration so he installed it and within a half hour he had it up and running. He created the graph as a version of the “stacky graphs” you can make in OpenNMS, but it was much easier to do and to maintain.

The name “stacky graphs” came from another customer of ours. They asked me if there was a way to put the bandwidth from all of their peer points on one graph. Now, in OpenNMS, it is easy to make a graph of data from a single device, and it is easy to group multiple graphs together, but it was not easy to put disparate data points on a single graph.

However, OpenNMS is a platform so I was able to find a way. When you create a graph definition in OpenNMS, there are two important fields, called “columns” and “type”. The “columns” value defines the file to look for, say ifInOctets.rrd and ifOutOctets.rrd, and the “type” value tells OpenNMS where to look for those files. So what I did was create symbolic links under the OpenNMS node directory named things like LAX-in.rrd, LAX-out.rrd and NYC-in.rrd, NYC-out.rrd that were linked to the interface RRDs of interest. Then I created a report of type “nodeSnmp” with column names like “LAX-in, LAX-out, NYC-in, NYC-out” etc. Then I could use AREA graphs to print out the data.

This was a pain for a number of reasons. First, you had to do a lot of configuration on the command line. Second, sometimes it is useful to delete .rrd files that haven’t been updated in awhile, but if you aren’t careful you’ll delete the symlinks. Finally, it is a lot of work to add new data sources.

Grafana Graph vs. RRDtool

In this picture you can see the Grafana dashboard in the lower left corner and the OpenNMS “stacky graph” in the upper right. Not only does the Grafana version look better, it will be easier to maintain moving forward.

I am eager to see what others are doing with this, so feel free to check out the integration on the wiki and let me know if you come up with anything cool.

Open Source Software and Corporations

Posted on by Tarus

An interesting post caught my eye this week entitled “Corporations and OSS Do Not Mix” by Ian Cordasco. It was kind of depressing – here was a person who had spent a lot of free time contributing to open source code, but the actions of some users of that code had taken the fun out of it.

My only issue with it was the targeting of “corporations” in the title. At OpenNMS we have a large number of corporate customers and we get along with them just fine. I want to talk about that in a bit, but first I want to address some of the other experiences Ian had that were similar to mine.

When I became the maintainer of OpenNMS back in 2002, I would often get e-mails from people that would start out with “OpenNMS is good, but what you need to do is …”. I used to spend a lot of time responding to them, pointing out that it was open source and anyone can help contribute to it, so they didn’t have to wait on me to do anything, but it never really helped and it turned into a huge time suck. I started to send back a generic e-mail that went along the lines of “OpenNMS is an enterprise product and if you won’t take the time to understand it then you should try something easier like Nagios” which would usually result in a reply calling me an asshole, but it took little of my time and then conversation was over. Now I pretty much just ignore them.

When you create something and share it, you are putting a bit of yourself out there and there are bound to be critics. For the most part they can be ignored, and you have to develop a thick skin to be in this environment. I’ve found that overall the good far outweighs the bad, and if you can learn to brush off the bad you can be very happy working in open source.

People tend to forget that open source “business” is still “business”. People exchange money in return for services. If I had Ian’s talent I would simply set up various custom development options, so when someone complained about a bug he could just return an e-mail with a price list. If you don’t have time to do it, make the prices really, really large – large enough that you would make time to do it. It’s your life – you are in the driver’s seat. I used to give a talk on running an open source business and I always stressed that you should never compete on price, or at least you shouldn’t lead with “my solution is cheaper”. Sure, open source software can provide tremendous savings over the life of the solution, but that doesn’t mean the solution itself is inexpensive to get set up. Done right, it will be better than any proprietary solution, but that doesn’t mean it comes without cost.

Always remember: free software does not mean free solution.

Getting back to dealing with corporations, like any interaction between two parties is it extremely important to set up expectations. You need to clearly outline what the product the client is buying covers (response time, 24/7 support, etc.). If they aren’t buying anything, then you don’t need to worry about them. I chuckled when I read “Well if you’re not going to take this seriously, we’ll have to start using another project.” We often get the “use another project” line and my response is “knock yourself out”. If you want to take this seriously, then pay me for my work. It’s like going into a free kitchen and complaining the soup is too salty.

A more difficult issue comes when someone wants to submit substandard code. This does require a little effort, since you can’t be sure that this isn’t just an eager but inexperienced coder versus someone lazy. Again, expectations are important. If you publish what the base level of quality should be, such as “must include unit tests”, then you can point to that when you don’t accept a submission. Plus, git makes it very easy to track a master branch and just apply your changes, so some sort of reply about how to do that could deflect criticism about the speed in accepting pull requests.

Ian makes a lot of really good points in his post, but I think he misses a point that if you run your open source project like a business then corporations (i.e. other businesses) will respect you and treat you like a business. We have one amazing company that just hired four (!) OpenNMS developers to work on code that they need. While some of it, if not most of it, will address their particular needs, all of it will be put into OpenNMS and they are paying us (gasp) to help project manage that team. That relationship did not happen overnight, but was built on a series of successful projects where we delivered particular value in exchange for money.

Look, I love, by and large, the open source community and I like being a part of it, but that doesn’t mean that open source and business are mutually exclusive. Learning to deal with open source as a business not only insures more open source gets created, but it also keeps it fun.

OpenNMS RMI Exploit

Posted on by Tarus

Recently, my RSS feed on OpenNMS stories turned up an article listing a possible remote code execution exploit in a number of applications, including OpenNMS.

In it, the researcher shows that it is possible to execute code on the OpenNMS server remotely due to a bug in the Apache commons library, which OpenNMS uses.

We’re a little unhappy that they published this without letting us know first (note that the e-mail address “security at opennms dot org” exists for reporting such things), but it is pretty easy to make sure that your instance of OpenNMS is safe. Simply configure the server’s firewall to disable remote access to port 1099 (it will need to remain for localhost).

I was happy to notice that the example he uses seems to be related to OpenNMS running on Windows. It can be a bit tricky to get OpenNMS to work on Windows, and perhaps the Windows default firewall doesn’t block port 1099 so that it why they noticed it.

It is a good idea to run something like iptables on your OpenNMS server and limit remote access to a minimal set of ports. Technically, the only port you really need access to is 8980, which is the default port for the webUI. I would assume that you would want port 22 for ssh access (unless you want to use the console for all configuration). In addition, port 162 should be open for SNMP trap reception.

That should be it. Now the application needs access to other ports (such as 5817 for events) so those need to remain accessible from localhost (127.0.0.1 or ::1) but that limits all exposure to only people who have shell access to the server, which we assume you limit to those people you trust. Remember to include IPv6 firewall rules if you use it.

An easy test to see if that port is remotely accessible would be to run:

telnet [IP or hostname of OpenNMS server] 1099

from a remote system to see if you can access the port. No connection should be made.

Sorry about this, but as I mentioned this wasn’t revealed to us until after the exploit was public. We are looking in to how we can better protect against this issue from a code change standpoint, but until then simply blocking access to the port will prevent most problems. We do plan to have a code fix in place soon.

The Inverter: Episode 53 – They’ve Got a Flamethrower

Posted on by Tarus

Okay, so I’ve been slack at getting this review out, since by now they’ve already had the planning meeting for next week’s show. As they mention at the start of this one, both Jeremy and Jono were unavailable for the last planning meeting so Stuart and Bryan ran with it. It was a good show, but it kind of demonstrates that, like many of us, the guys are very busy and sometimes you just have to soldier on, which I think is a great set up for the quality of this blog post.

I’ve been traveling a lot and I’m about to head out again, in part, to attend two great open source conferences in Europe, but last week found me in Rochester, NY which was an easy drive to Buffalo, where I met up with a recovering Jeremy Garcia.

Jeremy Garcia at Buffalo Proper

Due to my fascination with classic cocktails, we ended up at Buffalo Proper, where it turns out they make great drinks. This was right after the taping of the show, so I heard a bit about it from Jeremy and then listened to it on the plane ride back home.

The first segment talks about all the new cool open source computing devices out there, and if they are just for über geeks or will they ever appeal to the masses. I love reading about all the new toys that are available, but unfortunately I’m so busy that I can’t ever find time to play with them. I bought a Raspberry Pi when it first came out, but after it sat on a shelf for six months I gave it away to someone who might actually have time to use it. It took me forever to get around to making an OpenElec/Kodi PVR and without a specific need it is hard for me to find time to just play. I think these things will become more popular, but it will take time as young people (who tend to have more free time) discover them and start coming up with ways to use them.

Think about Lego. When they just made generic sets of bricks, they were a well known company but not very large. Then they started making sets to build specific things, and the brand took off. We’re are the “generic brick” stage now, but I expect something to come along that will create a huge increase in what things you do with these devices.

I am often jealous of today’s youth. Back when I was in school we didn’t have the Internet, per se, but we did have access to a number of dial up services. I used to call into BBS systems a lot (mainly running WWIV) and even figured out how to dial in to the campus network and access the VAX (which was connected to the Internet). There I could use “talk” to communicate with friends. Now, kids today have access to orders of magnitude more information and more toys. Unfortunately, that comes with the risk of “cyber-bullying” and other problems, but still, for those so motivated the benefits outweigh those risks.

I was surprised they didn’t talk about the ruling by the Librarian of Congress that made it (more) legal to tinker with technology you buy, which I think is a great step toward opening up tinkering at all levels.

The next segment discussed “vigilante malware” which uses the same exploits as regular malware but does it in order to make things less vulnerable to attack. Is this a good thing? The guys all agreeded that having someone change things on your devices with out your permission was “bad”, but they differed on the level of bad. I take a different approach. I work hard to keep my equipment up to date, so my assumption is that I wouldn’t be affected. However, many geeks and most muggles aren’t so aggressive, and so they get owned. This results in things like my mailbox being hit by spam (I get around 150 spam messages a day – most caught and processed by our mail server). This wouldn’t happen if people were more careful, as most spam originates from infected PCs, so I’m all for vigilante malware. Think about it – malware isn’t going away so why not encourage more of the good kind? Think of it like “good” vs. “bad” cholesterol. The only real solution to both is better security practices and better code, and both types of malware are incentives.

I think there is a hole in my logic somewhere. It’s kind of like the joke that you should always take a bomb onto a plane. Because while the chance of there being a bomb on a plane is slim, the chance of there being *two* bombs …

Anyway, the third segment talked about the Owncloud application. I’ve been meaning to play with this for some time (see “no time to play” above) as it looks cool. Take all of the nice features of “cloudy” things like Dropbox, and put them on a server you control. I think this is a fine goal. Plus, Owncloud also includes calendaring and contact management (apparently). We currently use Sogo for that, but it would be neat to integrate that with other things.

The only thing that wasn’t clear to me was the business model. The founder Frank Karlitschek states that Owncloud is not “open core” (or as we like to call it “fauxpensource“) but I’m not clear on their “enterprise” vs. “community” features. My gut tells me that they are on the side of good. I can see having a different license for an “enterprise” feature such as Sharepoint integration, especially if Owncloud has to use a proprietary library in order to get it to work at all, and it doesn’t look like the “server” version is intentionally hamstrung in order to get more business. Only finding the time to play with it will let me know for sure.

The final segment concerned laws about open source. The thesis is that the open source community spends a lot of effort working against laws that limit open source, so why shouldn’t the proprietary software world have to fight against laws that would make open source the norm? From the example above, the Software Freedom Conservancy spent a lot of effort to get the Librarian of Congress to make an exception to allow you to examine the software in various devices you own – why shouldn’t other companies have to fight to keep their code closed?

I think the team got this one right – money. Proprietary software companies get an immediate financial gain when their lobbying efforts pay off, but it doesn’t work for free software. However, I am seeing in these days of cost cutting that there is a movement in some governments to promote open source, so I think it is more of a question of true education than lobbying. One of the issues is that it gets confusing when companies like Owncloud offer an “enterprise” version and it isn’t clear what that means. While it might be 99% open source, all a detractor has to do is say “look, Senator, you have to pay just like you do for our stuff, and you know our stuff”.

Overall, decent episode. I get a mention in the outro as Jono refers to Todd Lewis, one of the people behind the All Things Open conference, as the “Nicest Man in Open Source”. I once held that title, but I would happily cede it to Todd. He is a truly nice guy, and is always willing to give you a hug. I used hug too, until that time I hugged Jono in Munich and what happened next had to be explained to my therapist with dolls.