Month: May 2012

A Cautionary Tale: Rogue Amoeba and Apple

Posted on by tarus

At the OUCE people were teasing me because I kept saying “Apple is Evil”.

Now I really don’t believe in good and evil as absolute constructs, but the phrase gets my point across. If you are a casual user of Apple’s products you probably don’t care, or more likely just don’t notice, how tightly Apple is trying to control both the information you create and what you can do with it.

[Note: for those of you who point out that Apple is a fine example of capitalism, they aren’t. In functioning markets, profits like Apple’s are not sustainable because competition will arise to drive prices down, to the benefit of consumers. With the double edged sword of stupid software patent lawsuits and a hegemony on components brought about through having container ships full of cash, Apple is preventing this from happening, or at least slowing it down, but I guess that’s a topic for another rant].

Some people think that I hate the idea of paying for software. Not true, I spend a lot of money each month on software. I prefer free alternatives, but I am more than willing to purchase commercial software as long as the vendor looks out for my privacy and the software works (or better yet, I prefer spending my money supporting free software).

One company I’ve spent money with over the years is Rogue Amoeba. They write wonderful audio-focused utilities for OS X. Recently I used Audio Hijack to digitize a collection of vinyl albums and then processed them through Fission. Through their Airfoil products I could stream music to any of my devices (well, before I started using Ubuntu – I really wish someone would work to make Linux tools that can leverage Apple’s Airplay/Airtunes technology or create a free alternative). The apps were inexpensive, easy to use, and got the job done.

Yesterday on the Rogue Amoeba website, they posted the following: “Apple Has Removed Airfoil Speakers Touch From The iOS App Store”

Airfoil Speakers was a little app that allowed you to stream music from your computer to any iOS device. It had been around since 2009. No reason was given for why this application was pulled and nothing had recently changed to act as a trigger for its removal.

Rogue Amoeba isn’t some little flight by night software company. They’ve been around a long time, been very focused on OS X (even when it wasn’t cool) and are all around good guys. While they are probably somewhere down the list of Apple business partners from, say, Samsung, my guess is that they are much closer to the top than most. The fact that Apple would just yank their app with no justification shows how little Apple cares about those who develop on their platform.

Think about it – Rogue Amoeba, and thousands of other software companies – have bet the farm on being able to supply software for Apple devices. These are good companies employing smart developers who have mortgages to pay. Now, to a large extent, their livelihood is being threatened by Apple’s fickle control of its marketplaces.

The rumor is that Apple is creating a similar product in the next iOS release, and apparently the terms of service prohibit applications that duplicate Apple-provided functionality.

Control is the main point here. One reason I left Apple last summer was that I saw that company taking more and more control over what I could and could not do with the devices on which I create. I could envision some point in time where Apple would make a decision I didn’t like, and then it might be too late or too expensive for me to back away.

I could see Apple moving to standardize everything, including their personal computers, on an iOS platform. With terms that prohibit competing products to anything Apple itself creates they are stifling competition, and I could foresee a future event where their policies might arbitrarily screw me over, although I didn’t really have a concrete example.

I do now, even if it didn’t happen to me directly.

Nifty Trick for Auto-Acknowleding Notices

Posted on by tarus

OpenNMS has a built in notification system that can act like a mini-trouble ticketing system. It’s triggered by events. Once an event creates a notice it “walks” a destination path where various actions can occur (send an e-mail, send a page, call a phone number, etc.)

The path has various “targets” and there is an escalation delay between each one. If the notice goes unacknowledged, the next set of targets is triggered.

Each path also has an “initial delay”. This is some amount of time where no notices are sent. It is useful combined with a feature that allows certain events (specifically nodeUp, interfaceUp and nodeRegainedService) to automatically acknowledge corresponding “down” events.

I hate getting a page at 2am that something is down. What I hate more is when a minute later I get a second page that the problem has been resolved. To minimize alerting on these transient outages, I use an initial delay of two minutes (or more) for most of my paths. This allows OpenNMS to make multiple attempts to see if the problem is resolved before notifying me.

Now, there are some issues with the notification system. First, it is triggered by events. This can be a pain for, say, a pesky trap that comes in every minute. Those can be reduced using the alarm sub-system within OpenNMS, so the plan is to eventually move the notification system to trigger on alarms. Since we already have a trouble ticket API built in to alarms, rather than modify notifd we plan to move that functionality to a separate product in the future.

The second is that this auto-acknowledgement feature only works if each event can be refined with a nodeid, interface or service name. If not, there is no way to differentiate similar events, and thus the system may acknowledge the wrong notices.

I just finished working on a support ticket to implement this auto-acknowledgement functionality using alarms, and I thought I’d share my solution.

First the events in question have to become alarms. This is done by adding an <alarm-data> tag to each event. The down should look something like this:

<alarm-data
  reduction-key="%uei%:%dpname%:%parm[#1]%"
  alarm-type="1"
  auto-clean="false"/>

Note that I just chose an generic %parm[#1]% to use the first event parameter to uniquely identify this event, but it could be any parameter, or a combination of parameters, depending on the event in question.

The up would thus look like:

<alarm-data
  reduction-key="%uei%:%dpname%:%parm[#1]%"
  clear-key="[UEI of Down Event]:%dpname%:%parm[#1]%"
  alarm-type="2"
  auto-clean="false"/>

Note the addition of the “clear-key”. This must match the reduction key of the “down” event.

Once these alarms are in place, it is pretty simple to create an automation.

Edit vacuumd-configuration.xml and add an automation:

<automation name="ackMyNotices"
            interval="60000"
            active="true"
            trigger-name="selectMyUpAlarms"
            action-name="ackMyDownNotices" />

with a trigger and an action. The trigger should fire when the “up” alarm is generated:

<trigger name="selectMyUpAlarms" operator=">=" row-count="1" >
  <statement>
    SELECT *, now() AS _ts
    FROM alarms
    WHERE eventuei = '[UEI of Up Event]'
  </statement>
</trigger>

So if there is at least one “up” event, we’re good.

Finally, perform the action:

<action name="ackMyDownNotices" >
  <statement>
    UPDATE notifications
      SET answeredby='auto-acknowledged', respondtime=now()
      WHERE eventid = (SELECT lasteventid FROM alarms WHERE reductionKey = ${clearKey})
      AND answeredby is null
      AND pagetime <  ${lastEventTime}
  </statement>
</action>

This will update the entry in the alarm table to show it as being “auto-acknowledged” if it isn’t acknowledged already. The tricky bit is that join where the eventid from the down alarm is matched with the eventid used for the notice.

Note that this works best with alarms that aren’t reduced, i.e. there aren’t multiple downs before the up event.

2012 OpenNMS Users Conference Europe

Posted on by tarus

Schnapps is Evil.

That isn’t the only thing I learned at the fourth annual OpenNMS Users Conference – Europe (OUCE), but it was the last thing so it stuck in my mind.

But I’m getting ahead of myself.

After spending a great week in Montréal, I was home for around 33 hours before heading back out to the airport. Even though there is a direct flight from RDU to London, I try to avoid Heathrow whenever possible, so I flew two hours in the opposite direction to catch a flight from DFW to Frankfurt. While the flight was uneventful, I didn’t manage to sleep so you can imagine that I looked and felt my best upon arriving at 8am on Monday.

I met up with Jeff, who had flown in from Atlanta, as well as Ronny and Markus, and we headed to the OpenNMS office in Fulda.

Since it is associated with the University, we then headed to a student run place called Cafe Chaos for coffee and to catch up.

Over time a number of other OpenNMS folks joined us. Christian is working on our VMWare integration, and Dustin is one of our Google Summer of Code folks. Sven brought us a cool little plush toy representing the Planck Epoch, or the universe immediately after the Big Bang.

I’m not sure if it was to scale – those things are pretty hard to measure.

I was also introduced to Club-Mate, the official drink of European hackers. It’s an ice tea-like beverage with a high caffeine content and relatively low sugar. I needed caffeine at the time so I drank a half litre, and I can understand the origin of the drink’s tag line which is “You get used to it”.

After lunch we wandered around campus and looked at the facilities. Since Ronny teaches part time at the school, Markus’s impersonation of a young student calling out to him “Herr Trommer” in a high pitched voice became a meme for the week.

Due to a mixup in scheduling, the OUCE was scheduled the same week as a huge emergency service vehicle convention, so all of the hotels in Fulda were booked, thus we got rooms just outside of town at the Hotel Gasthof Rhönblick. It was actually quite nice for the price, and I always like to stay a place where bikers are welcome. That afternoon we checked in and then headed back to Fulda for dinner.

Monday evening we met up with Uwe from Nethinks and his bride Daniela. While the company was awesome at dinner, by this time it was getting close to 11pm and Jeff and I had been up for 34+ hours, so we called it a night.

On Tuesday the OUCE started with the first of two training days. I did a one day seminar on the basics of OpenNMS. While our formal training offerings are a little more structured and go into greater detail, this is a good introduction to the application. Roughly 90 minutes are spent on each major facet of OpenNMS: provisioning, events, service assurance and data collection. We had a little over 30 students in attendance and everyone seemed to enjoy it (I didn’t see anyone sleep or look to terribly lost).

Wednesday was the second, or advanced, training day. This was added after last year’s conference when people wanted to explore the more complex features of OpenNMS. Jeff and Ronny taught this day (to let my voice rest) and it included things like the remote poller, the trouble ticketing interface and JasperReports. We had a few more people show up to for this class since they had been there the year before, and the only complaint I heard was that it was a lot of material to cover. Even when some of the hands-on stuff is removed we are trying to squeeze a five day class into two so it can be a bit much to take in.

On Thursday the OUCE kicked off in earnest. We decided to split the two days into a standard conference agenda on Day One and then a Barcamp on Day Two.

I started the day off with a presentation on the State of OpenNMS, which included a review of the new features in OpenNMS 1.10 as well as the future direction as we build toward 2.0. I also was able to present a mockup of a new user interface based on Vaadin.

This was followed by a great presentation by Alex Finger on project management for large NMS deployments. As someone who has been creating network management solutions for 25 years I can’t stress enough the need for some formal project management structure in order to guarantee the success of such projects.

It was during this talk that I started to structure the agenda for the OpenNMS Users Conference Americas (OUCA) to be held at the end of June. While a lot of people involved with OpenNMS are technical, many tend to have more political than technical challenges to overcome, so I thought it would be cool to have a track focused on the politics of getting a solution like OpenNMS into an organization instead of just how to use it. Alex is going to give this presentation again in Minneapolis as part of that track.

The next four talks were focused on integration. Jeff and Ronny talked about integrating OpenNMS with Puppet. Markus did a very technical talk on instrumenting Java applications with JMX so that OpenNMS could be used to monitor metrics from inside the applications themselves. Michael Batz from Nethinks demonstrated how he was able to use NagViz, a tool designed to integrate with Nagios, to present data from OpenNMS in a graphical format (i.e. maps), and I closed the day with a “Stupid Data Collector Tricks” presentation that included, among other things, a detailed description of my OpenVND project where we used OpenNMS to monitor a drink machine.

After seeing how well these talks were received, I invited the presenters to repeat them for the OUCA. It kinda solidified the theme of “Integration” for the conference. Although, perhaps we’d have more people show up if we unpacked our suitcase of “happy cloud”.

At the end of my presentation we got together to plan Friday’s barcamp. Everyone got up and introduced themselves, and we took down suggestions for topics and placed them on the wall. People would then vote on the talks they most wanted to see, and the top talks would be presented.

We had a little time for a break before everyone headed to the local brewery for an evening event, and no, this is not where the Schnapps comes in. I really like the Wiesenmühle – the beer is excellent and the food is good – although I was unhappy that this wasn’t the right time of year for Dunkel.

After eating, I excused myself for a couple of hours so that I could head to the office and catch up on e-mails and phone calls. Two hours later when I went back there were some hardcore OpenNMS folks still at it, but our designator driver was able to drive everyone back to the hotel.

Friday we split the day up into two tracks. The first session found me discussing some useful Net-SNMP tricks while Ronny discussed some ideas for using OpenNMS to monitor IPMI interfaces.

The second session was combined to discuss developing the European user community. While it didn’t get a ton of votes, I felt it was important enough to warrant the slot, and considering that we went 30 minutes over the time budget, we had a lot of good discussion. I really, really want to see the development of an independent users group for conferences such as this. I have no problem with The OpenNMS Group acting as a sponsor, but this should be something for the users by the users and I am certain that our partner Nethinks, who carries the majority burden of organizing this thing every year, could use some help. This session resulted in the creation of a new opennms-europe mailing list and I’m hoping to see the discussion continue there for next year’s conference.

After lunch we had three more sessions. In the first one, Ronny talked about Christian’s VSphere integration for monitoring VMWare while Markus talked some more about JMX integration (it was a popular topic). After that I did a presentation on making OpenNMS highly available, while Jeff did a session on Drools integration. His talk went almost an hour over time (well into the afternoon break) so we shifted the last session out even more. In that one, Jeff talked about backup strategies and Michael revisited maps.

During the afternoon we said goodbye to a number of people who had trains to catch, and the overall atmosphere for the conference was very upbeat. People seem to like OpenNMS and are eager to get most out of it, which does require an investment in time and energy even if the software is free.

There was still a core group of us left after everyone else had gone, and by this time I was totally shattered. After being on the road for most of six weeks, and the last two being out of the country, I just needed to unwind.

This is where the Schnapps comes in.

We dropped our bags off at the office at went to a Cuban-themed bar for drinks. Things started off pretty simply, but I am certain I was the one who called for the first round of Schnapps. We relaxed and talked for a long while, and then the subject of dinner came up. Places were still busy, it being a Friday night as well as the other convention being in town, so we decided to head back to the Wiesenmühle for dinner.

And more beer. And more Schnapps.

I forget how much we drank that night. I remember Ronny teasing me around 10pm since that was when I wanted to be in bed, but although my memory is a bit hazy I’m certain that was a couple of hours before we actually stopped for the night. Both Ronny and Markus abstained so they could drive the rest of us back to the hotel.

I don’t really remember that ride.

I do remember getting to the hotel, saying goodbye to Ronny and then remembering, too late, that my laptop bag holding my room key was still in his car. Needing sleep, I found that the doorway to my room was comfortable enough. I then I remember sometime later being wakened by Alex who had managed to get the master key from the innkeeper to let me in my room (these locks used old school physical keys and not key cards).

The next thing I know it was 5am. I’m awake, on my bed, completely clothed, and all the lights are on in the room (I haven’t done that in decades). I set my alarm for 7am, turned off the lights and got undressed for a two hour nap before heading to the airport.

Ronny showed up Saturday morning to drive me, Alex and Ian (another attendee) to the airport. I never saw Jeff. My trip home had me going through Heathrow as I wanted to visit some friends in the UK, and I am happy to report that I managed to stay awake as Martin drove me from the airport to his home in Lyndhurst.

There, his wife Sue plied me with the restorative powers of proper English tea, and after what seemed like about two pots I was pretty much myself. On a walk around Lyndhurst I found the next company car, if I can get it with left hand drive.

Craig visited from Southampton and we had a nice dinner, and I managed my first real good night’s sleep in weeks.

On Sunday, Sue drove me back to the airport in her new Fiat 500 (what a fun little car) and I had an uneventful trip back to Raleigh (where I was happy to find two Global Entry kiosks at customs).

It was a great week, and I hope to capture some of that energy for the conference next month. We have a number of folks signed up already, and remember that the early bird special ends this week. Space is limited due to the capacity of the rooms we’ve rented from the University, so if you are thinking about coming, register soon or drop me a note so I can save you a space.

And remember, Schnapps is Evil.

Montréal

Posted on by tarus

I just finished a week in Montreal. This was my first time in Québec, and while the weather could have been better, I really enjoyed the trip.

Québec is like France but with subtitles.

The primary language is French, but many signs are also in English. The rule seems to be that the English translation has to be below, and in a point size no larger than 50%, of the original French. Hence, subtitles.

I stayed downtown and took the Metro out to the customer site. They are a managed service provider with a goal to manage more than 50,000 access points across the country. They had been using OpenNMS for awhile, so it was a lot of fun for me since I got to help solve interesting problems, but it was also a lot of work because I got to help solve interesting problems. They worked my brain pretty hard to come up with a way for OpenNMS to address their needs, not just from a functionality standpoint but to come up with solutions that could also be automated and scaled. It was amazing how flexible OpenNMS can be, and with one exception we were able to address everything without changing the code.

I can honestly say that they would not have been able to do this with any other product out there. Not only are other solutions too rigid, their cost struction (i.e. per node pricing) would make them prohibitive on cost alone. Score one for open source.

It wasn’t all work. I did get to sample some fine cuisine (Montréal has a large number of nice restaurants) as well as local favorites such as “steamies” (hot dogs) and poutin (fried, fresh cut potatoes covered in a gravy sauce and cheese curds).

We also went out to see a midnight showing of “The Avengers” (review: very good, but unless you are a huge comic fan, maybe not as great as some of the reviews suggest – and stay until the very end of the credits) so I”m beat.

Plus I only had 30 hours at home before heading to Germany for the OpenNMS Users Conference Europe.

(sigh)

I’m sitting at DFW as I write this, on my way to FRA. The trip started out poorly since I forgot both my jacket and my headphones (which usually live in my case but were in my jacket since I’d only been home for a short while and, hey, I’d obviously remember to get it on the way out) but I did manage to find an inexpensive jacket at the airport (thank goodness for the changing seasons).

But things started to get better as I managed an upgrade on the first flight, and as I walked into the Admiral’s Club I almost bumped into Snoop Dogg, who was heading out. I’m glad I didn’t as the enormous bodyguard with him would have crushed me like a piece of paper.

I’m looking forward to seeing everyone in Fulda, and I’ll try to keep the blog updated throughout the week. Remember that we are bringing the goodness that is the OpenNMS Users Conference to the US this year. Hope to see you there.

Upcoming OpenNMS Events

Posted on by tarus

The most exciting part of the year, at least from the perspective of OpenNMS, is now upon us. In the next two months we have a number of awesome events happening.

Next week is our fourth annual OpenNMS Users Conference Europe. There is
still time to register (until Friday) if you want to join us in Germany for four days of classes and presentations:

We have also announced our next week-long training class to be held at company headquarters in Pittsboro, NC, USA the week of 4-8 June.

Next, our seventh (wow) annual developers conference, Dev-Jam 2012, will be held in Minneapolis, MN, USA the last week in June.

If you develop on OpenNMS, this is a great way to spend a week with most of the major contributors and to sharply hone your skills with respect to the OpenNMS code.

And now for the big news, we are also announcing the first ever OpenNMS Users Conference Americas to be held for three days starting 26 June.

Modeled on the European conference, the first day will consist of a seminar (taught by yours truly) covering the basics of OpenNMS. The last day will consist of a barcamp. This is an “unconference” where the attendees set the agenda. We’ll entertain suggestions for talks, vote on them, and then present the most popular ones. With most of the development team in attendance, pretty much any topic is up for grabs and expect it to be addressed by experts.

The middle day is very special. It is a standard conference day that we would like to use to focus on our community. That means you. We’ll be giving a roadmap presentation first thing, but then we’d love to fill the day with presentations by you on how you use OpenNMS.

If you are interested, send an e-mail to the call for papers: cfp@opennms.org telling us what you’d like to talk about. If it gets accepted, you’ll receive a complementary conference pass. I wish I could cover the air travel as well, but we will cover the conference, room and board for our speakers.

Hope to see you in person soon.