Month: October 2009

The Kindness of Strangers

Posted on by tarus

So far the OSMC has been very enjoyable. The talks have been good and the networking outside the talks has been even better.

Dr. Michael Schwartzkopff did a presentation on SNMP, and his slides could have been lifted from some of my own. He, like me, is a fan of Net-SNMP, and I learned about a log monitoring feature that is pretty cool. He also took time to help me figure out how to get Net-SNMP to properly generate traps based on values that Net-SNMP can monitor natively (blog post coming soon). His company has a three digit Private Enterprise ID (813) whereas mine is in the low four digits (2432).

I also got to meet Kristian Köhntopp, who is anything but a fan of SNMP. He gave an interesting talk on monitoring MySQL, but he has also known our own Alexander Finger for many years. It was fun to watch a debate between Michael and Kristian on the merits of SNMP, although there was no clear winner (I had my favorite, however).

My own talk went okay. I got laughs where I wanted to get laughs, and I didn’t see anyone sleeping. Also, no one left the room, which I think is a first (grin). The only problem I had (outside of speaking at a Nagios conference) was with the funky headset they asked me to wear (it kept slipping off my big head).

After the talk I got to meet Jens Bothe from OTRS. Especially in Europe, OTRS is a very popular ticketing system. Jonathan Sartin has created a bi-directional interface between OpenNMS and OTRS that was committed back to the OTRS project. It is available as a package so it is easy to install.

The biggest surprise was meeting Rihards Olups. He is part of the Zabbix project, and over the last year or so I’ve been hearing more and more about Zabbix so I’m very interested in hearing his talk. One thing I like about them is that they seem very similar to us: a small group of dedicated people working to create both open source software and an open source business.

Unfortunately, also like us, they seem to be the victim of code theft, or at least some very questionable licensing practices. A company called Firescope appears to have appropriated some of the Zabbix code. They have raised several million dollars in VC funding, lead by a company called Technology Advisors, and it seems they have used this money to put a prettier front end to the Zabbix code (sound familiar?). Apparently efforts by Zabbix to get this issue resolved have been a failure.

Note: Please read the comments below. The CEO of Firescope has responded to these claims with more information. I am hoping that Alexei will respond when he can. When I get some time I plan to download the trial version of Firescope as well as Zabbix and see for myself if the similarities still exist, but until then I will withhold judgement and I ask that you do as well.

This really pisses me off, of course. I think part of the issue is that Zabbix is headquartered in Latvia, and my guess is those people at Firescope felt that they would not be in a position to challenge or question the Firescope use of their code. And the unfortunate part is that they are right. It’s a sad fact that very few people outside of a small open source community cares. We found out with our own copyright issues that there is little recourse when someone appropriates your code that doesn’t involved spending a lot of money with lawyers. Some people will point you to the Software Freedom Law Center (an organization I support with an annual donation, by the way) but the charter of the SFLC prevents them from helping projects with a commercial interest, even, as is the case with Zabbix and OpenNMS, if that interest is not a large, well-funded corporation.

We were lucky in that we have become profitable enough to afford proper legal council, and so we could hire Eben Moglen and his team at Moglen Ravischer. When I heard about the issues Zabbix are having with Firescope, I made an introduction between Alexei Vladishev and Eben’s team in the hope that they could work out an arrangement like we were able to do.

Some might find this unusual, since on the surface it might seem that OpenNMS and Zabbix are competitors. In fact, we have run into them at one or two accounts, so the smart thing to do would be to ignore their problems. It may be the smart thing, but it wouldn’t be the right thing. We learned a lot going through this process ourselves so it would be wrong not to help another company, especially one so similar to our own.

Anyway, I hadn’t really thought about this again until I met Rihards. He introduced himself and apologized that Alexei could not have come in person (apparently he is off doing great things with clients in Japan). And in a very surprise move he presented me with a ceramic magnet from Riga and a loaf of amazingly dense and sweet smelling rye bread.

I was touched, and we struck up a friendship. I’ve been told the best way to eat the bread is with cured meats (and me having just left Italy [sigh]) and good butter. I also have a standing invitation to visit and eat wild mushrooms (not the “magic” kind, however).

Rihards joined Ronny Trommer and myself at an event last night that involved a lot of beer, good food, and a surprising amount of American pop music from the 1990s. It was a fun night, and I look forward to the resolution of the issues between Zabbix and Firescope, as well as eating those mushrooms.

Tonight Ronny and I head off to meet Klaus Thielking-Riechert. He is putting us up for the night. Klaus, along with Ronny and Alex, are writing an OpenNMS book. And unlike the book I am writing, this one actually exists and I look forward to seeing it in print.

When you think about it, all great friendships start with a meeting of strangers.

RRDTool 1.4 Released

Posted on by tarus

I saw this morning that RRDtool 1.4 was released. While there are a lot of changes, the biggest one seems to be the addition of rrdcached. From the release notes:

The RRD Caching Daemon can dramatically improve the ‘update’ performance of your system. Due to file handling overheads, the time it takes todo one update is virtually the same as to doing two updates in a row.

The Cache Daemon intercepts rrdtool update calls, assembling multiple updates before writing them to the actual rrd file. When calling rrdtool graph in such a setup, the command will tell the daemon to flush out all pending updates for the rrd files, required to draw the graph.

What I think is funny about this is that Matt Brozowski added this same functionality to OpenNMS years ago. It has been the only way we have been able to keep up with data collection at our larger installations.

The next time I get access to a large environment, I’d love to measure the performance of OpenNMS running with JRobin and queuing versus OpenNMS running with RRDtool with rrdcached and no queuing. If anyone has time to play with this, please let me know the results. You will need to compile jrrd against 1.4, of course. Also, it doesn’t look like Dag Wieers has 1.4 RPMs yet, but my guess is that he will soon.

2009 Open Source Monitoring Conference

Posted on by tarus

I’m in Nürnberg, Germany this week to speak at a Nagios conference of all things. I was a little disappointed, however, to learn that Nürnberg is no where near the Nürburgring, but so far I am happy I came.

The conference is sold out, and the people I’ve met so far are pretty cool (although I haven’t met all of the 260 in attendance – yet). It’s an interesting ecosystem that has been built up around Nagios, and in many ways it is a microcosm for open source as a whole.

OpenNMS and Nagios are about the same age, but Nagios is much better known, in part because it is written in C and has been available in a number of distributions for years. OpenNMS, being written in Java, has always faced some resistance from users who just don’t like Java, and until the recent advent of the OpenJDK there as been a hurdle to getting something like OpenNMS included in major distributions.

Nagios and OpenNMS share some overlapping features, but from the beginning OpenNMS was aimed at large scale enterprises and was designed with that in mind. Nagios appealed to users at a smaller scale, who liked the ability to add almost any check they could think of to the system very quickly.

But there are more differences. Ethan Galstad, the author of Nagios, always kept pretty close control of the core software. As I write this, OpenNMS has over 40 people with commit access to the code, while Nagios has 9 (up from 3 just a few months ago).

This has caused some tension and has resulted in a number of new projects based on Nagios and at least one outright fork. I can count at least two here at the conference: Opsview and Icinga, and while Ethan used to attend this show he is not here this year.

Maybe it is a coincidence, but last night he announced Nagios XI. This appears to be an “open core” version of his monitoring product, with the old Nagios now being called “Nagios Core OSS”.

As you can imagine, I’m not excited to see more open core software. Groundwork has raised nearly US$30 million and continues to fail with its open core fork of Nagios, and while I consider Ethan much more capable I think it is a bad move. The popularity of Nagios was driven by its community, and it seems obvious that this community is not happy. Driving another wedge between them by splitting development into open and closed versions will not help.

But this is the beauty of open source software. It doesn’t matter what I think. The users get to make the decisions, and Nagios has a great group of users.

Perhaps I can get a few of them interested in OpenNMS.

Dual Licensing and the Beauty of Positano

Posted on by tarus

Note: As usual, I have over-analyzed an issue that normal people wouldn’t spend two minutes thinking about. I apologize to my three readers in advance, but unlike more famous commentators on open source I often find that it is not possible to explore the nuances of issues in our community with a couple of short sound bites.

I have been in Italy for the last week scouting locations for our Italian headquarters. I’m thinking Positano would be a great place:

I was also able to take a few days off for a short holiday, but I couldn’t stay off the Internet for long. As I was skimming through my news feeds, I came across a post by Brian Aker about dual licensing open source software, and it stuck in my mind. It wasn’t necessarily the topic but it was the, well, harshness of his post. It was full of absolutes and hyperbole which is very unusual coming from Brian, and it must have struck a chord with others as it managed to appear on Slashdot as well.

For a little background: Brian was one of the early contributors to MySQL, and MySQL pioneered the idea of “dual licensing” open source software. The legal basis for open source is based on copyright law (distribution of software is considered in the same fashion as distributing “copies” of other works, such as books and music). The holder of the copyright is able to set the terms by which copies can be made. MySQL published its code under the GNU General Public License (GPL) which allows for derivative works to be created and distributed as long as the requirements of the license are met. By its very nature the GPL makes it difficult to commercialize the code itself, since when it is distributed one is required to include the source.

However, MySQL retained 100% of the copyright to its code, and so it was also able to offer it under a license that was different than the GPL. Thus if a company wanted to embed MySQL into its product but didn’t want to be subject to the GPL, they could purchase the rights to use the software under a proprietary license. This enabled MySQL to create a revenue stream from the code, which allowed it to hire more people, advertise, etc.

In my very first post on the subject of open source licensing and business models I mentioned this model as being an acceptable way to fund open source development. The main caveat was that there may be some resistance from the community to contribute code if they had to transfer the copyright to MySQL, but it didn’t appear that the MySQL community cared.

Well, times change. It is my understanding that as MySQL grew they started to offer code to their commercial clients ahead of the community, and they may have even adopted more of an “open core” model where some of the code was never offered under an open source license (I’m not 100% sure of the details). Eventually, MySQL was bought by Sun and now Sun is in the process of being purchased by Oracle.

And this is where the trouble begins. MySQL was one of the first, if not the first, open source database projects that aimed to provide an alternative to commercial products such as those sold by Oracle, so one has to wonder what Oracle will do with MySQL. It is doubtful that a company like Oracle will focus on undermining its main product line with an open source alternative. I’m not saying that they will kill off the product as I seriously doubt that, but the motivations behind it have changed (see the 451 Group for a better timeline on this).

This has resulted in some turmoil in the MySQL community and the announcement of a number of code forks to MySQL. At issue is that since Sun, and possibly now Oracle, holds the copyright to the code there are some limits placed by the GPL for anyone wanting to commercialize those forks or even create free alternatives. Brian Aker works on a fork of MySQL called Drizzle, and I believe that his involvement in, and possible frustration with, the whole copyright issue lead to his post on “The Peculiar Institution of Dual Licensing“.

He starts off with a quote from Richard Stallman that ends:

With excellent management and considerable trust within the user community, MySQL became the gold standard for web based FLOSS database applications.

I think the key phrase here is “considerable trust”. In the case of MySQL, there must have been some considerable trust within the community to contribute code since it became the property of MySQL. More on this in a moment.

To move on to Brian’s comments, the first sentence that struck me as odd was:

Dual Licensing is nothing more then the creation of yet more software, where the end result is the creation of not more open source software, but the creation of yet more closed source software.

I had to think about this one. The only way I could make it work was that if open source code is available under a dual license, then that code can become part of closed source software through the purchase of a commercial license. That code is by definition closed source, since there would be no reason to purchase a license if it was open source code. But my guess is that any real open source software company would use the money raised from such licenses to focus on the main open source product, thus increasing the amount of open source software. Also, if the “Bubba’s Inventory ‘n Stuff” application chooses to license and embed MySQL, its distribution will probably be more limited than MySQL itself, so the ratio of open to closed software is relatively unchanged and may even increase with respect to open source.

Brian continues:

The GPL’s approach is to provide a stick or carrot. If you are open source, then you don’t pay, assuming of you are “the right” sort of open source. If you are close source or pick a license which is not compatible with the GPL you are forced into paying for use of a commercial license. When you “pay” for open source the freedom that was originally offered to the end user is removed.

I don’t understand this talk of “‘the right’ sort” of open source. It’s like the guy who compared the GPL to DRM. The creators of open source software have a wide variety of licenses they can use, from the very permissive Apache and BSD licenses to the more restrictive GPL. Brian seems to be making the case that some are better than the others. He used the word “forced” – who is “forced” to use open source software? If you don’t like the license, don’t use the code. It’s not about forcing people to use software but offering a choice. Dual licensing actually increases freedom – you have the freedom to use the software at no cost under the open source license, and you also have the freedom to purchase the right to use it outside of such a license.

Recently I’ve been giving a talk about what it is like to run a business based on open source software. While the presentation is constantly evolving, it starts off with the two main options for creating powerful open source software. One is to form a “foundation,” preferably non-profit, that is funded by companies with a vested interest in seeing a certain type of application created. This has resulted in a number of amazing projects such as Firefox and Apache. A company like IBM may want to have an alternative to Microsoft Office without having to own such a product line, so they fund OpenOffice. Rackspace may need the benefits of a project like Cassandra, and so they keep people on the payroll to make it a reality. In many cases these projects are published with very permissive licenses so that the commercial entities behind them can take full advantage of the software without many restrictions like those imposed by the GPL. Since permissive licenses allow easy integration into closed source software, I’m not sure if Brian would consider this the creation of more of the same, but in any case it is hard to argue that certain projects have thrived very well under the foundation model.

The other way to produce large scale open source software is to create a company to support the project, such as is the model with OpenNMS and The OpenNMS Group. In this case funding the effort is more problematic. Without a “patron” supplying the resources like in the foundation model, a business has to look more seriously at the bottom line. It has to build up a customer base and a brand, and in the case of a bootstrapped company like OpenNMS this takes time. Despite that we have created some really amazing code, but without a more restrictive license like the GPL someone with money could take our work and quickly commercialize it. It is important that the ownership of OpenNMS is controlled by a single group to enforce the license and to prevent this. Brian has his own comments about ownership:

At the heart of dual licensing is “ownership rights”, which inevitably come into conflict with the nature of open source. Open Source projects that preserve the ability to do dual license come into conflict with the developers who contribute the code. For the project to continue it must ask the original developer to give up their rights to the code via copyright assignment (there is some debate on whether copyright can be held in joint, but this is often disputed by lawyers). Thus dual licensing forces any developer who wishes to contribute into a position of either giving up their rights and allowing their work to end up in commercial software, or creating a fork of the software with their changes. In essence it creates monopolies which can only be broken via forking the software. Forking software over small changes is for the most part unviable because of the cost of keeping a fork of the software up to date, but it is not impossible.

I really dislike this paragraph since it tries to paint in shades of black and white a subject that is anything but. Since the entire concept of open source software is based on copyright law, by its very definition it requires an “owner” to enforce the copyright. At OpenNMS we found out the hard way that when the copyright is held by multiple parties it makes it very difficult to enforce any license.

I also don’t think that it is “inevitable” that these rights will cause conflict. Brian is probably experiencing some friction between Drizzle and MySQL, but I don’t think that one example makes conflict “inevitable”.

He then brings up my original criticism of MySQL’s contributor agreement which required that a developer assign all rights to any contribution to MySQL. Think about it – suppose some person came up with a cool algorithm that improved performance and contributed to MySQL, assigning all rights. This would not only give MySQL complete ownership of the work product, but it would prevent the developer from using his work in another project.

This is why at OpenNMS we adopted the Sun Contributors Agreement which allows for multiple copyright holders. Contributors assign copyright to the maintainers of the project, which allows them to more easily protect it from abuse (as well as dual licensing the code if they so chose), but they retain the copyright to their work. For example, Bobby Krupczak contributed a data collector for XMP, a new network management protocol he is developing. He assigned copyright to that work to OpenNMS so we could use it in our project but he retained the rights to his work so he could also contribute that code to another project or create one of his own. The contributor loses no rights yet the project can maintain some form of control over the whole work. I think this is an awesome concept but Brian states that “this is often disputed by lawyers”. He doesn’t link to a single example and while I am certain there are one or two lawyers who would be willing to raise an argument against it, in my experience it isn’t a concept that is “often disputed”. It reminds me of the fear, uncertainty and doubt spread around the GPL and its enforceability, but as time has shown the GPL does stand up in court and I think the idea of dual copyright will as well.

In addition, dual licensing provides an option for companies that would like to use open source software and perhaps even contribute to it but for a variety of reasons cannot. One recent example would be Apple’s use of ZFS. It was obvious that they were considering support for ZFS but for some reason they changed their mind, and the rumours indicate licensing. As it would have been in Apple’s best interests to contribute back to the project, this would have been a win for ZFS, but now that option for improvement is gone.

I think that Brian is reacting more to a problem specific with MySQL than with open source as a whole. His frustration is evident with such loaded phrases as “take the rights from others” and “lack of imagination”. I don’t believe anyone’s rights have been taken away, except for those that freely gave them to MySQL. If there is some assumed right that open source code can be used by anyone at any time for any reason, then we are no longer talking about open source. The creators of open source software have always been able to protect both their work and their rights through a variety of means, but there has never been a real or implied right granted to end users that wasn’t specified in the license. It seems Brian wants to move the slider from “open” to “free” but I don’t think that is sustainable in many cases.

What I do think is sustainable is a 100% open source project that also has a dual license option. It does involve a level of trust between the copyright holder and the community and the implied covenant that profits from such licenses will be fed back into the project, but it is definitely doable. In this instance Stallman got it right.

Win.

Secretary of State FAIL

Posted on by tarus

One of the many things I do at OpenNMS is handle the finances. This often requires me to interact with the Federal and State governments. To say that the system could be simplified is an understatement.

For example, take this postcard I received in the mail. It was for a corporation that was dissolved in 2003, but for some reason the office of the North Carolina Secretary of State (with whom I dissolved the corporation) still thinks that it is, in some form, active.

As you can see, the card tells one not to contact the office of the Secretary, but instead to call a particular number to talk with the Department of Revenue.

Go ahead, dial it. I dare you.

It appears that the Department of Revenue didn’t pay their phone bill, and the number has been disconnected.

(sigh)

Sometimes movies like Brazil get it right.

Mercedes Profitable

Posted on by tarus

A few weeks ago I commented on a Paul Graham post about being “Ramen Profitable” with a reply that at OpenNMS we were “Sushi Profitable“. I also speculated that one day we would be “Mercedes Profitable”.

Well, I wouldn’t say we were there yet, but I did buy a Mercedes.

To be honest, I didn’t pay all that much for it (I bought it from friends) and a C230 isn’t exactly the best example of the marque (as this “Overhead in New York” points out) but I like it.

I can’t see spending a lot of money on a car. I mean, I could pay myself a bonus or get a company car, but we try to plow all of our profits into making the company stronger. One way to do this is to hire more people and, as we are having a good year, I am happy to announce that Jason Aras has decided to join our team.

Jason has been involved with OpenNMS for years now. He is a member of the Order of the Green Polo and can often be found on the IRC channel as “fastjay”. He even came out to Germany, on his own nickel, to present at our first OpenNMS Users Conference. We are very excited to have him on board.

I am often asked how one gets a job working on OpenNMS, and the best way is simply to show us what you can do by getting involved. It may sound a little self-serving, but we’ve been successful by attracting people who really enjoy what they do. It is the second part of our mission statement: Have Fun. If you wouldn’t spend some of your time on the project “for fun” you probably wouldn’t enjoy it as a job, either.

When I was in college the big thing was to co-op. The co-op program let you take time off from school to work, preferably in a field you were hoping to enter, both to earn some money and to see if you liked it. I think open source provides an even better opportunity, as it allows people to really hone their skills, both in things like programming as well as working in a team, through the simple act of showing up. It can provide experience that one can put on a resumé which shows initiative, talent and the ability to work remotely and with others.

And who knows, there might even be a paying job in it.

Roadmaps

Posted on by tarus

The roads in San Antonio are something to behold. Immense concrete leviathans that snake above the ground, surrounded on both sides by access roads that are superhighways unto themselves. My GPS goes crazy because it is simply not accurate enough to tell if I’m on the frontage road or on the highway above.

Think I’m kidding? In downtown on I-10 they actually have signs indicating “upper level” and “lower level”. That’s right, they built an interstate on top of the first interstate. I almost wrecked looking at it, as it is quite the engineering feat. The upper deck is cantilevered way out from the supports. It’s amazing.

I was downtown to meet up with Eric Evans. Eric is also amazing: Debian maintainer, OGP Emeritus and a self-taught developer who has probably forgotten more about code than I’ll ever know.

I met Eric back in April of 2002 when I first started doing work for Rackspace. He is now working on the Rackspace Cloud, where Rackspace has dedicated full time resources to the Apache Cassandra Project (I knew I loved these guys for a reason).

During the meal the topic of roadmaps came up. Someone on the Cassandra IRC channel had asked about a roadmap and Jonathan Ellis replied with a list of things they had targeted for the next few releases. This didn’t sit well with the questioner. He wanted something a little more solid, with firm dates, etc.

The answer was: that doesn’t exist. The development team is dedicated to moving Cassandra forward, but needs and priorities are very fluid. They are more focused on doing new, useful, and interesting work and not in meeting some sort of artificial deadline.

[Note: this is my interpretation of the story and may not accurately reflect the thoughts of Eric, Jonathan or others within the Cassandra project.]

This reminded me a lot of how OpenNMS is developed.

We always have at least two releases of OpenNMS going at any one time. The stable, or production, release experiences as few changes as possible and most new code is designed to address bugs. The second number in the version is always even: 1.0, 1.2, 1.6, etc.

Then there is the unstable, or development release. The term “unstable” has nothing to do with the robustness of the code, but that whole chunks might change from release to release, which in turn can introduce bugs (although the main goal is that any outstanding bugs in an unstable release are minor or cosmetic). The second number in the version is always odd: 1.1, 1.3, 1.5, etc.

We have various feature targets for each release. For example, 1.8 has two main features: a new provisioning system and access control lists. Thus 1.8 will be done when those features are complete. When we believe that the main coding has been done for 1.8, version 1.7.90 will be released. Chances are there will be a 1.7.91, 1.7.92, etc. Once we are happy that it is worthy of production use, it will be christened 1.8.

There is one more feature coming in 1.8: documentation. OpenNMS is really an amazing piece of software worthy of the name “network management application platform” but more than half of the features are undocumented or at least underdocumented. This will change.

So, when will 1.8 be released? The smartass answer is “when it is done” but the truth is I don’t know.

There is one other factor in the mix, and that is the commercial company behind OpenNMS. While we have a very active community, most of the code comes from people who get paid to work on it, simply because they can work on it full time. With our business plan of “spend less than you earn” we have to focus on the bottom line, and a good portion of that income is the result of custom programming projects. Right now we are in the middle of two big projects with two more waiting in the wings, and thus we are pretty much at capacity when it comes to working on the main features in 1.8. This doesn’t mean we’re not working on them at all, but other things take precedence so it goes more slowly than I would like.

I’m sure there are people out there who will advise me that this is the wrong way to go about it. You need a firm roadmap, they’ll say, and they may be right. But I have to rationalize what I want with what I can afford to do, and something has to give. I could focus all of our resources on getting 1.8 out, but then there might not be a company around to support it when it is done. Plus, one of the things I go on and on about is the flexibility of open source software, and if I am not able to provide custom solutions to clients how true can that statement be?

There are others who might say that this focus on client code isn’t “open source”. Fully 100% of the code we write is published under an OSI approved license. Heck, you can even see what we are up to by browsing the subversion repository or subscribing to the opennms-cvs mailing list. We are completely transparent in that regard.

Finally, we plow all of our profits back into the project by hiring more people. I’ll have an announcement along those lines next week.

Some might say that the lack of a roadmap means less releases and thus less press to entice new users. That may be true as well, but at least we’re at a point where a large number of people don’t even know OpenNMS exists and, once they do, find that 1.6 is more than adequate to address their management needs.

So, to summarize: yes, we have a roadmap for OpenNMS but it can change over time depending on custom development we get paid to write. The upside is that most of that development is of benefit to other users, and often comes with a timeliness that a static roadmap couldn’t match.

The OpenNMS roadmap evolves in a way that both addresses the needs of our clients and keeps us in business. That’s a nice road to be on.