I spent last week (the last of my three week trip to Europe) in the German city of Köln (Cologne). I’ll probably keep referring to it as “Köln” since I just figured out how to easily print special characters on my Ubuntu desktop. (grin)

I was working with a service provider called NetCologne, specifically on a project where they are rolling out a Fiber to the Building (FTTB) project. Ronny Trommer, one of our guys in the European office, did most of the heavy lifting, but I was there to help out (at least in the beer drinking capacity).

We needed to monitor and collect data from a growing number of remote devices, and when I last checked we were polling 26110 distinct devices and 51690 interfaces. One thing that came in handy with this deployment was the Instrumentation Log Reader.

If you haven’t seen this feature, you should check it out. Under Admin there is a menu item called the “Instrumentation Log Reader”. This will generate a very useful report from the instrumentation logs concerning how many devices are being scanned for data collection, how long it takes each scan to complete, the time period between scans and the number of failures.

At NetCologne with 300 threads (the default is 50) the system was not able to poll fast enough to provide five minute samples, so we needed to increase the number of threads. While it was not easy to do a weighted average, it seemed that a conservative estimate would be about 10 seconds per device to perform collection. There were some of them in the 20-30 second range, and quite a few below 5 seconds, but for calculation purposes 10 seemed like a good number.

This meant that each thread could collect on six devices a minute, or 30 devices per five minute period. Since we needed to collect on 26110 devices, that meant we needed a little over 870 threads. We made it 1000 just to be sure, and then data collection started performing well.

A lot of the time was spent integrating their internal systems with OpenNMS through the provisioner. Much like Towerstream, any service provider with this many devices needs to be able to automate adding and removing devices to their management system. I am certain that the strength of our provisioning system is the one major reasons large enterprises and carriers should consider switching to OpenNMS.

On my last night in Köln I was taken out to a brewery near the Cologne Cathedral, or Dom, to partake in large amounts of the local brew called kölsch.

I found out on Google Plus that a number of people have very strong opinions about the quality of that particular beer, but I liked it. It was very light and clean, which was helpful in that we drank quite a bit of it. It came to our table in a “ring” of glasses:

and we went through several rings (I lost count).

No matter what you may think of kölsch, I don’t think anyone can deny the beauty of the cathedral:

Even at night covered in part by scaffolding, it is a gothic, fractal masterpiece.

I had fun, and I look forward to the opportunity to go back as this project progresses.

It worked out that after the training week in Fulda, a new client of ours wanted to start the first week of their Greenlight Plus project, so I just stayed in Europe another week.

Headquartered in Vipiteno, Italy, the Leitner Group is one of the two largest producers of “ropeways”, what we would probably call ski lifts and gondolas. Through a number of mergers, their products have grown to include snow grooming equipment, other passenger transport (through their Sigma subsidiary, they built the cabins for the London Eye), as well as small wind turbines (with the awesome name of Leitwind).

It was because of this recent growth that OpenNMS was needed to replace their Nagios instance that just couldn’t be used to manage the 600 devices spread out across 60 or so locations. It was a fun project, made even more so by the fact that Antonio Russo, our go-to man in Italy, worked on it with me.

I am planning on writing up a more detailed post on the transition, but it was pretty straightforward. OpenNMS was used to replace Nagios, Cacti and Smokeping, and with only two exceptions we were able to replace the Nagios “check scripts” with built-in OpenNMS monitors. For the ones we couldn’t migrate we just used the General Purpose monitor and ran them as external programs. Since Nagios runs all of its checks as scripts, it must fork-exec each one which severely limits its scale. All of the OpenNMS monitors run inside the OpenNMS process itself (well, with the exception of the GpMonitor), and the fact that you can also do data collection and event management without extra integration steps was a bonus. Antonio took the Nagios configuration files and ran them through a script which automated moving the devices into OpenNMS using the provisioner, so it was pretty simple to get the nodes into our application.

Working at Leitner was fun for a number of reasons. First, the technology is cool. If you have ever been on a ski lift you know that the speed of the cabin changes when it reaches the ends so that people can get on and off, and this is usually accomplished by having the cabin disconnect and then reconnect from the main cable (which moves at a constant speed). The equipment that does this is pretty complex, not only due to the size and weight of the components involved, but also due to the safety concerns (don’t want to be dropping your paying customers).

Second, working with the client, Michele, was a lot of fun. I’ve often bragged about the fact that OpenNMS users are self-selecting. We don’t go out of our way to market the application, so the people who find it, download it, install it and decide that they want to spend money on services are pretty special. They are all amazingly bright people, friendly and witty, and without exception tend to be very good looking.

(grin)

Third, Vipiteno is an incredibly beautiful place. Nestled in the Italian Alps, it is a very clean and walkable town. It has a split personality, since for much of its history it was located in a German speaking country. Like Canada, almost all of the signs are in two languages, and when I am talking about it to Germans I have to refer to it by its German name of Sterzing.

The whole week I had to keep reminding myself that I was actually at work – getting paid to have this much fun seems criminal.

On Wednesday, Antonio drove us down to Bolzano to visit some of his friends. As you cruise down the highway, it was not unusual to see castles – old school, Holy Grail, storybook castles.

Whenever you saw one, you often could look across the valley and see another. I keep wondering if they ever went “Game of Thrones” on each other, and thought about the logistics of waging war in this terrain.

Yes, I’m a bit odd, I realize that.

Unfortunately, it got dark so I couldn’t enjoy much of the countryside. Most of the castles were illuminated with exterior lights, however, so they seemed to float in the air as we sped past.

Bolzano was a cool place – a much larger city than Vipiteno. One of Antonio’s friends works for the hockey team, so we got to see them practice.

According to Wikipedia, the Bolzano-Bozen Foxes “is the most successful team in the history of the Italian ice hockey league, with 18 championships.”

It was a fun visit, although we stayed out so late that it made getting up on Thursday a lot harder.

Thursday night we visited another city, Trento, about 50 kilometers further away than Bolzano. It was a business meeting with a potential client, and I hope to be able to talk about that project soon. Driving back it started to rain, and thus we woke up on Friday to see snow on the mountains.

Friday was my last day at the office (we had to drive back to Frankfurt that afternoon), so we spent the morning finishing up the migration. When it came time for lunch, I was expecting that we would take Michele out to a restaurant, but he had a surprise for us. We walked over to his house and met his family, and his wife cooked me the best meal I’ve had in Italy this trip.

Michele speaks at least three languages, his wife is fluent in five, and even his oldest son (who I think is around four or five) can speak three. I had a very nice conversation with him in English. Michele’s youngest son was sleeping, so I didn’t get to spend much time with him, but even at two he’s probably studying ancient Greek and Relativity. (grin)

It was a great ending to a wonderful week.

As I’ve mentioned before, there is something about the Open Source Way that brings vendors and clients closer than usual. I have even grown hate that term “vendor”, as I much prefer the term “partner”.

Or better yet, “friend”.