Eight Years and TriLUG

Last Thursday, 10 September, marked 8 years since I started working on OpenNMS.

Time flies.

That evening I presented a talk called “So, You Think You Want to Start an Open Source Business” at the Triangle Linux Users Group. TriLUG is the only LUG I know of that holds its meetings at Red Hat HQ, which I think is pretty cool. Unfortunately, it is rare that I can make a meeting, but I do try when I have the opportunity.

At each meeting they serve pizza, and since they know that Papa John’s is a big OpenNMS client, they bought from them (plus, they also knew that I was sponsoring it, so that might have had something to do with it, too).

I think the talk went well, and I plan to present it again at this weekend’s Atlanta Linux Fest, although it clocked it at over 90 minutes so I’m going to have to work hard to get it into my allotted 45.

I really enjoyed being back at TriLUG. When OpenNMS was just me, my social life was centered on TriLUG and its IRC channel (sad, I know). In some way they helped keep me going, so I feel a certain amount of fondness for the organization.

Which is why I was surprised to learn that TriLUG has fallen on hard times financially. There used to be enough in the bank to not only cover pizza during months without a sponsor, but to cover the travel expenses of speakers from out of town (which enabled a slew of interesting people to speak at the meetings). When I found out about this I promptly signed us up as a “Gold” sponsor.

I also want to challenge any of the local companies around here who benefit from Linux and open source to help sponsor this organization. It is quite possible the next big open source application could be nurtured by this group.

The UK's National Health System

I just finished some rare rest and relaxation over the weekend playing bridge with some friends in Oakland, California (for the record, 42.5 rubbers with a few slams and one grand slam that was made but not earned).

Over the table we started discussing the decline in rational discourse over the years. It seems that in the quest for attention, every discussion rapidly devolves into personal attacks and rhetoric. I’ve been a victim of this many times when trying to start a discussion of open source software, but my experience pales compared to what I am seeing in politics.

I don’t know why this seemed less in the past. Perhaps the media did a better job of calling it out and making people stop. The hot topic du jour is the debate over government sponsored health care in the US.

A great example of the problem was illustrated by my favorite news anchor, John Stewart. He contrasted video of Glen Beck trumpeting that “America has the best health care system in the world” with video from a little over a year ago where Beck experienced that system and was, shall we say, considerably less than satisfied with it.

It’s worth a look.

One health system that is commonly trotted out as an example of what not to do is the one in the United Kingdom. Now this blog is not the place to discuss health care but I felt that, as one of the few Americans who have experienced the British NHS first hand, I should relate my story on something that could “end democracy in America as we know it”.

I live in rural North Carolina and there are lots of deer around. Hardly a day goes by that I don’t see three or four, and as a consequence I am exposed to a number of deer ticks – little disease carrying, blood sucking insects.

While I take steps to minimize this exposure, I still manage to get ten to twenty ticks bites a year. They’re no big deal – more of a nuisance really – but I do keep an eye on each bite just to make sure everything is okay.

In the spring of 2006 I got bit by a tick, removed it, and forgot about it. A few days later I found myself in the UK to do some work on OpenNMS. One morning I noticed a large, target shaped rash around the tick bite, which is a classic symptom of Lyme disease.

Being the digital age, I took a picture of it and e-mailed it to my physician back home. He replied that it was definitely Lyme disease and I needed to get a prescription for antibiotics. Since this requires a doctor’s involvement in the UK, I had to ask my client to help me get an appointment.

Now, this was about 11am on a Thursday. They made some calls and I was given a 1pm appointment at the local “surgery”. Since I wasn’t covered by NHS insurance, I was told I’d have to pay the “consult fee” out of pocket, which was £60, about US$100 at the time.

I walked in and was asked to sign a piece of paper with my name and address on it. That was it. I was ushered into an examination room promptly at one and the doctor arrived immediately after that.

She asked me to take off my shirt and in an instant recognized that the rash was due to Lyme disease. Then she asked if I would mind if she had the medical student look at it, since the New Forest, where I was, does have deer and the very rare case of Lyme disease. I had no problem with that and when he came in, he got excited and asked if he could take some pictures. I said “sure” and while he went to get his camera, the doctor got on the phone to call in the neighboring physicians. After the photo shoot and doctor parade, she even asked if the triage nurses could come in and take a look.

I was very patient through the whole thing and as a reward they waived the fee. They said it was the most interesting case they’d had in six months. I got a prescription and had it filled around the corner at the “chemist”.

I’m telling this story for two reasons. First, I like to joke that I once got paid $100 to take my shirt off (and for those of you who have seen me you realize why this is funny). Second, I hope this story will help counter some of the fear, uncertainty and doubt being spread about the whole issue of health care and specifically attacks aimed at the NHS. I found the whole experience as pleasant as one could expect, and on par with, if not better than, my average trip to the doctor in the US.

I can only imagine what a foreigner without insurance would have to go through in order to get similar care here.

More Adventures in Open Source

A friend of mine pointed me to a blogger in the UK who also calls his blog “Adventures in Open Source”. Sweet! The more the merrier. I’ve been promised a pint next time I’m in Liverpool.

The more people who get excited about open source the better. Perhaps it will help prevent more of this:

This is a picture of the monitor in the elevator at the Marriott in downtown San Francisco. It works about half the time.

I see a lot of this in my travels. I have yet to see a public screen displaying something like a Linux kernel panic, however.

2009 OpenSource World – Day 2

Not to start this post off on a down note, but wandering around OpenSource World I couldn’t help but think I was attending the last LinuxWorld Expo.

I’m not sure where the show lost its way, but the once huge conference was just a shadow of its former self. OSCON has become the main show for open source, and with Linuxcon coming up in a few weeks I can guess why the hardcore Linux crowd stayed away.

For example, the Linuxcon keynotes will include Linus Torvalds and Mark Shuttleworth. The first keynote at OpenSource World was a replacement speaker from Dell who simply went through their standard cloud computing slide deck. The hall was so empty they removed about 200-300 chairs from the back before the next speaker to keep it from looking so sparse.

Not a good sign.

I did actually have some fun at the two sessions I attended. The first was Brian Aker’s Drizzle talk.

As most of you know, Drizzle is sort of a return to the MySQL roots. It is designed to be a small and extremely fast database for use on web sites (yes, I’m oversimplifying so check it out if you need more). Being sort of a ground-up re-imagining of MySQL it tries to take the best of the older platform with an eye on the future. For example, Drizzle is being designed solely for 64-bit architectures on machines with lots and lots of RAM, the assumption being that once the project is ready for production that will be the default system anyone would want to use for it.

What I really enjoyed was his strategies for getting the community more involved in the project. He related a story of when MySQL started growing and they realized that everyone in the community who was a serious contributor had been hired by the company. I hope to adopt some of his ideas for community involvement in Drizzle for use with OpenNMS.

The second session was a panel hosted by Larry “Dark Lord of Open Source” Augustin on open source compliance. As Chris Dibona says “All Panel Discussions Suck” and this one was no exception.

The thing I hate about panel discussions is that there is little give and take between the audience and the panel members. For example, there was a lot of talk on concerns of using open source inside an organization with license compliance, compatibility and things like that, when the truth of the matter is that as long as code doesn’t leave an organization, open source software is pretty close to public domain. The rules don’t usually kick in until it is distributed (as the license is normally enforced under copyright law) and so many organizations can see huge benefits from taking and using open source software internally.

But I wasn’t able to ask about that.

Two of the panelists seemed to have a real solid understanding of open source software. Tim Golden, of Bank of America, brought up something near and dear to me: work for hire.

At OpenNMS we make a large portion of our revenue from custom development. All the work we do gets put back into the product, so the client gets exactly what they want out of OpenNMS and we get to make OpenNMS more powerful.

However, most standard contracts have a “Work For Hire” clause that states the client owns all the code, so we have to either remove it entirely or rewrite it extensively to make it more in line with the ideals of open source software.

He also brought up this scary concept of “residual knowledge”. It would seem logical that if I am doing work for a client and I come up with an idea during that time that I should own the idea. But what if that idea was triggered by some confidential knowledge I obtained from the client? Do they have any claim to future work based on it? Apparently this is a hot topic within cloud computing since previously confidential business practices get more exposure. It was a term I had not heard of before and one that I plan to keep an eye one (plus adding a new “residual knowledge clause” to our contracts).

I also was impressed with last minute panel replacement Steve Wretling, from Kaiser Permanente. Kaiser seems to have a real grasp of how to deploy open source within their environment and they have put in place processes to take advantage of it.

So, overall, while I hated the format I enjoyed the session.

The next keynote was by Debra Bowen, the California Secretary of State. She is a good speaker and very tech-saavy, but she has also been branded a Luddite due to being outspoken in her criticism of electronic voting machines. The stories she told were scary, and it made me value the optically scanned paper ballots still used in parts of North Carolina.

Then the fun began. I met up with John Mark Walker, Michael Coté and Luke Kanies for lunch. John Mark is starting a new venture promoting open source on Windows, Coté was unusually quiet, and this was the first time I’d met Luke.

I think I know why Coté was quiet.

Let’s just say if you ever need to kill a couple of hours, put me and Luke in a room and press “Go”.

I found Luke to be quite charming, and we found a lot of things in common between his team’s creation of Puppet and our team’s work on OpenNMS. We both can talk at length about a variety of subjects. But we are also very different: he’s skinny, I’m fat; I eat animals, he does not, etc.

Based on my experience with the morning panel discussion, we thought that ours should have more audience interaction and I think it went okay. We had a few people walk out and Luke says he saw one person sleeping but for the small crowd that stayed (and stayed awake) I hope they got a little something out of it. I had fun.

That evening Luke and I met up at the View Lounge at the top of the Marriott and continued our discussions on everything from open source, running a business, popular culture and shoes. We were joined by filesystem hacker extraordinaire Valerie Aurora and later by Michael Coté.

Val came up with a game to name three topics of conversation that should be banned when geeks meet socially. Last year her three were computers, the election and the financial crises. The new three became computers, health care, and social networks.

Hence the discussion about shoes.

After awhile we split up and Coté and I continued over sushi and sake. I always enjoy talking with him because he is an analyst who used to be a programmer. I think he gets where I’m coming from. The best technology salesperson I know is a guy named Doug Gilkey who has an advanced degree in aerospace engineering, and in much the same way Coté is one of the best analysts in the management space since he’s been there.

All in all I’m glad I came, but I don’t think I’ll be back. I can’t make Linuxcon this year, but perhaps next, and OSCON is apparently returning to Portland so I might have to combine a trip to that show with visiting clients in the area.

But I gotta buy some new shoes.

Spring in San Francisco

Okay, so I get on a couple of planes, watch five episodes of The Wire (finished Season 4) and land in San Francisco only to hit the VMWare/SpringSource news blizzard. As I assume my three readers know, yesterday VMware acquired SpringSource for around US$420 million – a nice chunk of change and one of the top five largest open source acquisitions of all time by my count (including MySQL, JBoss and XenSource in that mix).

I don’t understand it.

Don’t get me wrong – I’m not saying that SpringSource wasn’t worth the price. Heck, we use Spring technologies throughout OpenNMS and love it. We had Ben Hale out to our Dev-Jam conference way back in 2006 (when they were called Interface21) so Matt recognized the potential even then, but it just goes to show you how much I don’t know about this whole investment thing.

Let’s examine this deal. Matt Asay reports that the annual revenue at Spring Source was US$20 million. That means that VMWare paid a multiplier of 21 for the company (I believe 6-8 is much more common). That is amazing.

Second, Matt also reports that most of that revenue was in services. Huh? I’ve been told for 7 years now that services companies aren’t worth anything. In fact, I was told that by Benchmark, the people that just made a bunch of money on this sale. Tricksy, no?

One thing I do understand now is the acquisition of Hyperic. Not only did it benefit the VCs (the investors in Hyperic were pretty much the same as those for SpringSource) it brought extra revenue and products to the table. I am certain that Hyperic was responsible for a good portion of that multiplier.

Which makes me happy for the Hyperic team. While they were my favorite whipping boy for their unique use of the term “open source,” I know a few of the founders and they are genuinely nice people. It’s good to see them do well, and to Javier, Doug and Charles – next time, drinks are on you. I’ll eat a little crow as long as it is served with a nice Scotch.

Since I obviously don’t understand the world of mergers and acquisitions, I’ll stick with what I do know – enterprise management, helping customers and having fun doing it. OpenNMS is not only an application but a framework on which management applications can be built, and that seems to be pretty hot right now. Business is booming, but while I don’t see any US$500 million acquisition in the near future, check back in a couple of years.

We might just surprise ya.

Google the Alphabet

I’m a big fan of Google (who isn’t) but I’m often curious as to how page rank, etc., actually works. OpenNMS has been the first hit for “open source network management” for years, and I’m not sure why that is. We obviously don’t get much press or spend a lot of money on marketing, but we’re even ahead of the Wikipedia article in rankings.

In researching this post, I also couldn’t help but notice the number of other tools that have adopted the term “enterprise-grade”. We were the first to use that term (after we successfully deployed at a client site with well over 50,000 devices using a single instance of OpenNMS) and we still have most of the top ten hits under “enterprise-grade network management”. I assume they use the term because it sounds good, as I have yet to see anyone come close to OpenNMS in scale, especially when you divide by the number of servers needed. I can’t think you can call yourself enterprise-grade if it takes 100 servers just to manage the network.

Anyway, that really wasn’t the reason for this weekend post. As I assume everyone has noticed, awhile back Google added some nice AJAX-y completion features. Type in “open” and it suggests “open office”. I decided to see who owned each letter of the alphabet. Here is the list:

Amazon
Best Buy
Craigslist
Dictionary
Ebay
Facebook
Gmail
Hotmail
IRS
JCPenny
Kolhs
Lowes
MySpace
Netflix
Orbitz
Photobucket
Quotes
Realtor.com
Southwest Airlines
Target
USPS
Verizon Wireless
Walmart
XM Radio
Youtube
Zillow

Interesting, huh? Only two of the letters, D for Dictionary and Q for Quotes, are generic words. The government is represented with I for IRS and U for USPS. Online resources are prominently represented: Craigslist, Ebay, Facebook, Gmail, Hotmail, MySpace, Photobucket and Youtube. There is also one “.com” site with Realtor.com. The rest are all companies.

I had to look up Photobucket and Zillow, as I had never heard of those before (one is a Flickr-type site and the other is real estate focused). Three of them: Best Buy, Southwest Airlines and Verizon Wireless actually return two words even though only one letter was typed in the search box.

I’m not sure what it means to “own” a letter of the alphabet on Google, but it can’t suck.

Sushi Profitable

Paul Graham is one of the people whose advice I always welcome, even though I don’t always agree with it. Most of the time, however, he is spot on.

This week he posted an entry called “Ramen Profitable“. Ramen profitable means a startup makes just enough to pay the founders’ living expenses.

There was a time in the life of OpenNMS when we were ramen profitable. Heck, when I first started out I paid myself solely on commission, which means I was profitable from Day One (and believe it or not, that first year I only spent $5000 out of savings to survive).

Paul writes:

Another advantage of ramen profitability is that it’s good for morale. You feel like you’re finally earning your keep. A company tends to feel rather theoretical when you first start it. It’s legally a company, but you feel like you’re lying when you call it one. When people start to pay you significant amounts, the company starts to feel real. And your own living expenses are the milestone you feel most, because at that point the future flips state. Now survival is the default, instead of dying.

I love that last line: survival is the default. It’s something I’ve been trying to explain about our business model (spend less than you earn) but I haven’t been as succinct. Profitability gives you options. Profitability, especially in open source, means you have the time you need to build your product and your community, and you can remain true to both.

But as Paul mentions in the footnote, ramen is one of the cheapest foods out there. It is one thing to survive and another to survive in comfort. So I’m going to call us “sushi profitable”.

While we don’t pay the highest salaries out there, we are able to pay good salaries. While we work on used office furniture, we can afford new laptops. And if any one of our employees wanted to go out for sushi, they could without hardship.

Those of us who work on OpenNMS every day realize it is something special, and so we are willing to sacrifice immediate financial gain in order to help the company grow faster. That doesn’t mean that we aren’t comfortable. I’m not sure what the next phase after “sushi profitable” will be (Mercedes profitable?) but I’ll let you know when we get there.

Purists and Pragmatists and Zealots, Oh My!

I didn’t make it to OSCON this year (thanks for everyone who voted for us for the CCA, by the way, even though we lost out again to Firebird) and I am quietly thankful for it, because it seems like the conference kicked off a new round of hyperbole and hypocrisy from the fauxpen source crowd and I’m going to try to stay out of it (instead of any kind of rational discussion, this round seems even more full of ad hominem attacks).

I’ve been labeled both an open source purist and a zealot simply because of my assertion that the term “open source” is defined by the open source definition. And while no one calls me a pragmatist, only a pragmatist could have kept a company like OpenNMS going through good times and bad without investment.

Heck, I’m even pragmatic about open source – in the realm of enterprise network management nothing works better, but that doesn’t mean it works for everything.

But no matter how successful we are, someone will think we aren’t successful enough. Luckily, I haven’t spent much of my life worrying about what others think, and we have been so busy lately that I can easily lose myself in helping our customers and our community.

However, at the risk of boring my three readers, I wanted to share an epiphany I had at dinner Thursday night.

We were discussing open source and I reexamined why a number of commercial open source companies see so little contribution that they resort to calling their users “leeches” and free loaders.

It is just natural that in a gift economy like open source, those who give back in a substantial fashion will be few. I’m am always grateful for any positive contribution while having the expectation of none. OpenNMS enjoys a wide range of contributors, enough so that I would never feel the need to refer to our community as leeches. I was wondering what was so different about us from other companies.

The answer came to me straight from Dan Ariely’s fine book Predictably Irrational. In it he talks about “Social Norms” versus “Market Norms” with an example from a day care center in Isreal:

A few years ago, [Uri Gneezy of UC San Diego and Aldo Rustichini of the University of Minnesota] studied a day care center in Israel to determine whether imposing a fine on parents who arrived late to pick up their children was a useful deterrent. Uri and Aldo concluded that the fine didn’t work well, and in fact it had long-term negative effects. Why?

Before the fine was introduced, the teachers and parents had a social contract, with social norms about being late. Thus, if parents were late — as they occasionally were — they felt guilty about it — and their guilt compelled them to be more prompt in picking up their kids in the future. (In Israel, guilt seems to be an effective way to get compliance.)

But once the fine was imposed, the day care center had inadvertently replaced the social norms with market norms. Now that the parents were paying for their tardiness, they interpreted the situation in terms of market norms. In other words, since they were being fined, they could decide for themselves whether to be late or not, and they frequently chose to be late. Needless to say, this is not what the day care center intended.

Vibrant open source communities operate under social norms. I am often approached by people who say that they really love OpenNMS, but then they sheepishly admit that they don’t buy support or contribute in any other way. They tend to smile when I say that I’m cool with that – just finding our work useful makes my day and they’re probably not a good fit as a support client in the first place.

But the open core realm operates under market norms. If some restaurant is handing out free food samples, rarely does one feel guilty about taking some. In much the same way, when users see a company that sells commercial software, there is no obligation associated with taking the free stuff and being done with it.

So the problem of leeches is one of their own creation, and as research has shown, once you set the basis for interaction based on market norms, it is hard to move back. This may be one of the main reasons a least one such company seems to be changing its message away from open source.

Once again, I’m just thinking out loud, and in that vein let me state that I’m thankful OpenNMS still seems to be operating under social norms. I’ll work hard to keep it that way.