One of my favorite times of the whole year is the week of Dev-Jam, the annual OpenNMS Developers Conference. This year will mark our twelfth meeting, and it has grown quite a bit since our inaugural one in 2005.

For the first time we will be holding Dev-Jam outside of the United States. About a third of the attendees come from other countries and due to recent changes in US immigration policy we couldn’t have people forced to reveal sensitive things such as social media passwords just to come to Dev-Jam.

So, we are holding it at Concordia University in Montreal, Quebec, Canada. Not only is Montreal an awesome city, Concordia is also the alma mater of Jesse White, one of the key architects of OpenNMS. It should be a wonderful venue for the conference.

We have reserved a block of rooms in the historic Grey Nuns Building. Similar to the dorms we have used in the past, every one will get a single room with a shared bathroom. If you would rather stay in a more conventional hotel, there are a number of excellent choices nearby, although I would strongly recommend you consider the dorm as the majority of people will be staying there and it can be quite nice.

There are also a number of other events going on in Montreal that week, including a Metallica concert and a fireworks competition, and we will try to do something as a group (baseball is out since the Expos moved to Washington, DC, in 2004).

Space is limited, so if you are the slightest bit interested please let me know and I can reserve you a spot. More details can be found on the wiki and registration is now open.

Hope to see you there, and yes, there will be poutine.

The last day of Dev-Jam is always bittersweet. The bitter part is the goodbyes, but the sweet part is “Show and Tell” when folks share what they have accomplished in the week.

We also get together for a group picture. Just before that Jonathan’s son Eddie joined us from the UK on the robot:

and David, who had to leave for a family issue, joined us via robot as well.

All of the presentations are up on Youtube.

Chandra has been working on adding provisioning detectors to the Minion:

Deepak and Pavan, who work for a large electronic medical records company, discuss how they are using OpenNMS at scale:

Seth has been managing a lot of that work, which is currently focused on syslog, and he did a presentation on new syslog parsing functionality:

Alejandro presented some awesome improvements to the UI:

Markus has been working on project Atlas, which includes major improvements to OpenNMS maps. Here he demonstrates the integration of the geographical map with the topology map:

More UI enhancements were offered by Christian who added trend lines to the OpenNMS home page:

Ronny talked about his ideas for making device configurations more modular and managing that with git:

And he has also been creating reusable Docker containers with OpenNMS:

One project I found extra exciting was “Underling” which is an instance of Minion written in Go. This makes it incredibly small (about 6MB) which should allow the Minion to run on very inexpensive hardware.

I plan to demonstrate more Minion stuff at the OpenNMS Users Conference (and if you haven’t registered, you should).

In the evening we walked back across the river to dine at Town Hall Brewery.

It will be the last time all of us were together until next year, and I can’t wait.

Dev-Jam is made up of two main groups of people: those who work on OpenNMS full time and those who don’t. For those who work on OpenNMS full time, we try to depart from the day to day running of the project to both try new things and have fun. Think of it as “special projects week”.

Since OpenNMS is aiming to be a platform for the Internet of Things, this tends to involve a lot of electronics.

I decided to take some time out to further explore the Virtual Reality provided by Google Cardboard. I played with it last Dev-Jam, but I bought a nice headset from Homido since the Cardboard experience with the actual cardboard holder, while novel, was a little bit wanting.

The downside is that it doesn’t have the little magnet thingie that acts as a mouse click. Most people using the Homido tend to pair some other controller to their Android device in order to navigate, and since I have a PS3 (that I mainly use to play Blu-ray disks) I had a Sixaxis controller I could use. I had to buy an app in order to deploy a driver on my Nexus 6P that would work with the Sixaxis, and after a bit of tinkering I got it to work (note that it disables the regular Bluetooth driver when you run it).

I configured the “X” button to act as a mouse click, and pretty soon I was able to move about the Google Cardboard demos. The Homido fits well and the image is good, but it does allow some light to bleed around the edges in so it would work best in a dim or dark room.

I then went off to find some apps. This is not a field that a lot of developers have explored, and most of them are pretty passive. While this can work (check out the creepy “Sisters”) I wanted something more along the lines of what I experienced with the Samsung Gear VR, which includes immersive games. I found one called Hardcode VR that was fun, sort of a platformer along the lines of Portal. The controller worked out of the box exactly like you would expect it to: the right joystick was used for looking around and the left one for moving. I did get a slight headache after playing it for awhile, though, so I think that for the time being mobile device-based VR is still a novelty.

My experiment did amuse some of the folks at the conference, and Ronny made this comparison:

I am always humbled by the people who give a week of their lives up to come to Dev-Jam, and even more so since DJ was away from his wife on his birthday. We did make sure he had a cake, though.

The cake was from Salty Tart and it was mighty tasty.

It’s hard to believe this year’s Dev-Jam is half over. After months of planning it seems to go by so fast.

One of the goals I had this week was to understand more about the OpenNMS Documentation Project. For years I’ve been saying that OpenNMS documentation sucks like most open source projects, but I can’t say that any more. It has actually become quite mature. There is a detailed installation guide, a users guide, and administrators guide and a guide for developers. Each release the docs are compiled right alongside the code, and it even rates its own section on the new website.

It’s written in AsciiDoc, and all of the documentation is version controlled and kept in git.

Ronny Trommer is one of the leads on the documentation project, and I asked him to spend some time with me to explain how everything is organized.

Of the four main guides, the installation guide is almost complete. Everything else is constantly improving, with the user guide aimed at people working through the GUI and the administration guide is more focused on configuration. For example, the discussion of the path outage feature is in the users guide but how to turn it on is in the admin guide.

There is even something for everyone in the developers guide (I am the first to state I am not a developer). One section details the style rules for documentation, in great detail. For example, in order to manage changes, each sentence should be on a single line. That way a small change to, say, a misspelled word, doesn’t cause a huge diff. Also, we are limited as to the types of images we can display, so people are encouraged to upload the raw “source” image as well as an exported one to save time in the future should someone want to edit it.

It is really well done and now I’m eager to start contributing.

Speaking of well done, Jonathan has figured out what is keeping OpenNMS from using the latest version of OTRS (and he’s sent a patch over to them) and Jesse showed me some amazing work he’s done on the Minion code.

We’ve been struggling to figure out how to implement the Minion code since we want to be able to run it on tiny machines like the Raspberry Pi, but since OpenNMS is written in Java there is a lot of overhead to using that language on these smaller systems. He re-wrote it in Go and then uploaded it to a device on my home network. At only 5.6MB it’s tiny, and yet it was able to do discovery as well as data collection (including NRTG). Sheer awesomeness.

Wednesday was also Twins night.

For several years now we’ve been going as a group to see the Minnesota Twins baseball team play at Target Field. It’s a lot of fun, although this year the Germans decided that they’d had enough of baseball and spent the time wandering around downtown Minneapolis.

At first I thought they had the right idea, as the Braves went up 4 to 0 in the first and by the top of the fourth were leading 7 to 0. However, the Twins rallied and made it interesting, although they did end up losing 9 to 7.

Our seats were out in left field, ‘natch.

By Day Two people have settled into a rhythm. Get up, eat breakfast, start hacking on OpenNMS. I tend to start my day with these blog posts.

It’s nice to have most of the team together. Remember, OpenNMS is over 15 years old so there is a lot of different technology in the monitoring platform. I think David counted 18 different libraries and tools in the GUI alone, so there was a meeting held to discuss cleaning that up and settling on a much smaller set moving forward.

In any case ReST will play a huge role. OpenNMS Compass is built entirely on ReST, and so the next generation GUI will do the same. It makes integrating with OpenNMS simple, as Antonio demonstrated in a provisioning dashboard he wrote for one of his customers in Italy.

They needed an easier way to manage their ten thousand plus devices, so he was able to use the ReST interface to build out exactly what they wanted. And of course the source is open.

Several years ago we started a tradition of having a local restaurant, Brasa, cater dinner one night. This year it was Tuesday, and it is always the best meal of the week.

As we were getting ready to eat, Alex Hoogerhuis, a big supporter of OpenNMS who lives in Norway, decided to join us via our Double Robotics robot, Ulfbot. It worked flawlessly, and he was the best first time driver we’ve had. Ben, Jeff and Jonathan joined him for a picture.

We like using the Yudof Hall Club Room for Dev-Jam for a number of reasons, one includes the big patio overlooking the river with picnic tables. Alex was able to drive around and spend some time with the rest of the team, although we had to lift him up to see over the wall to the Mississippi (we also had to carry him in when the wind picked up – heh).

After dinner people kept working (DJ was up until nearly 2am chasing a bug) but we also took a break to watch Deadpool. It’s why “Dev-Jam” rhymes with “fun”.

Dev-Jam officially started on Monday at 10am, where I did my usual kick-off speech before turning it over to Seth and Jesse who handle the technical side of things.

Yesterday I stated that this was our tenth Dev-Jam at UMN. I forgot that the first one was held at my house, so this actually makes this the ninth (we’ve still had eleven since 2005).

Everyone went around the room and talked about the things they wanted to work on this week. A lot of them focused on Minion, a technology rather unique to OpenNMS. A Minion is a a Karaf container that implements features for remote monitoring. It is key for OpenNMS to be able to scale to the Internet of Things (IoT) level of millions of devices and billions of metrics. And speaking of IoT, Ken turned me on to openHAB which is something I need to check out.

It is often hard for me to describe Dev-Jam to other people, as it is truly a lightly structured “un-conference”. In a great example of the Open Source Way it is very self-organizing, and I look forward to Friday when everyone presents what they have done.

We did have Alex Finger, one of the creators of the OpenNMS Foundation, join use via robot. He was having some sound issues and I think he did get stymied by the robot’s lack of hands when he came across a door, but it was cool he was able to visit from Europe.

We use this week for planning and sharing, so Jesse took some time to go over the Business Service Monitor (BSM) which allows you to create a “business level” view of your services versus just the devices themselves. It is fully implemented via ReST and it pretty powerful, although as with a lot of things OpenNMS that very power can add complexity. I’m hoping our community will find great uses for it.

That evening about half of us walked to a theatre to see Star Trek Beyond. Most of us disliked it and I posted a negative review, but it was fun to go out with my friends.

♬ It’s the most wonderful time of the year ♬

Ah yes, it’s Dev-Jam time, where we descend onto the campus of the University of Minnesota, Twin Cities, for a week of OpenNMS goodness.

This is our eleventh annual Dev-Jam and our tenth at UMN. They are really good hosts so we’ve found it hard to look elsewhere for a place to hold the conference.

This is not a user’s conference. That is coming up in September. Instead, this is a chance for the core contributors of OpenNMS, and those people who’d like to become core contributors, to get together, share and determine the direction of OpenNMS for another year.

This year we are just shy of 30 people from four different countries: The US, the UK, Italy and India. Alejandro and his wife Carolina are now permanent residents of the US so I can’t really count them as being from Venezuela any more, and that happened directly through his involvement with Dev Jam. We’ve had more people but 30 seems to be the magic number (one year we had 40 and it was much harder to manage)

My trip to MSP was uneventful. I flew through Dallas even though there is a direct RDU->MSP flight on Delta since I’m extremely close to Lifetime Platinum status on American Airlines. Also, AA has added a cool feature on their mobile app that lets me track my bags. This was important since I was shipping a box of four 12-packs of Cheerwine – a Dev-Jam favorite and as always a target for TSA inspection (apparently a 40+ pound box of soda is suspicious). Everything got here fine.

Including Ulf:

Ulf is the OpenNMS mascot and he, too, is a product of Dev-Jam. Many years ago Craig Miskell came to Dev-Jam from New Zealand. He brought this plush toy and gave it to the Germans, who named him “Ulf”. Since then he has been around the world spreading the Good News about OpenNMS, so it wouldn’t be Dev-Jam without him.

We stay in a dorm called Yudof Hall where we take over the Club Room, a large room on the ground floor that includes a kitchen and an area with sofas and a television. In the middle we set up tables where we work, and due to UMN being a top-tier university we have great bandwidth. There is a huge brick patio next to it that looks out over the Mississippi River. It’s a very nice place to spend the week.

Speaking of the Mississippi, we crossed it last night to our usual kick-off spot, the Town Hall Brewery. As a cocktail aficionado, I was happy to see some craft cocktails on the menu, and a number of us tried the “Hallbach”, their take on the Seelbach Cocktail:

It was very nice, as they used a high proof bourbon and replaced the champagne with sparkling cider.

We like Town Hall since we can seat 30 people. We do cater in as well as go out. The new light rail service to campus makes getting around easy, especially to the Mall of America and Target Field.

Speaking of baseball, we’re all going to the game on Wednesday. If you are in the area and want to join us, I should have a couple of tickets available. Just drop me a note. We also brought along the Ulfbot, which is a tele-presence robot so do the note dropping thing if you want to “visit”.

Dev-Jam!

Yay! We have settled on dates for the eleventh (!) OpenNMS Dev-Jam Conference.

Once again we will descend on the campus of the University of Minnesota for a week of fun, fellowship and hacking on OpenNMS and all things open source.

Anyone is welcome to attend, although I must stress that this is aimed at developers and it is highly unstructured. Despite that, we get a ton of things done and have a lot of fun doing it (and I’m not just saying that, there’s videos).

We stay at Yudof Hall on campus, and while that can scare older folks I want to point out the accommodation is quite nice and I’ve been told they they have recently refurbished the dorm. If you want to stay on campus the cost is US$1500 for the week which includes all meals.

If you prefer hotels, there are several nearby, and you can come to the conference for US$800.

Registration is now open and space is limited. If you think you want to come but aren’t sure, let me know and I’ll try to save you a space. We’ve sold out the last two years.

Oh, sponsorships are available as well for $2500. You will help us bring someone deserving to Dev Jam who wouldn’t ordinarily get to attend, and you’ll get your logo and link on www.opennms.org for a year.

Dev Jam!

Sorry for the week delay on this post, but this happened.

The last day of Dev-Jam is always bittersweet for me. I’m sad that it is over, but I also get to see all the wonderful “new shiny” people have been working on. Friday we do demos.

This year we made an attempt to record each demo. Just click on the picture to see it on the YooToobz. Videos, yay!

First up was Ben. Ben is the architect of our new mobile app, OpenNMS Compass. Available for both iOS and Android, it may even turn into our next overall user interface. To do that, it needs graphs, so Ben demonstrated how you can now display graphs in Compass. You can even set “favorites” so they show up on your main screen.

Markus von Rüden spent the week working on something fun: digitizing our mascot and kiwi overlord, Ulf. He demonstrated this work in a game. While it wasn’t completely finished, when Ulf died would he split in half to reveal a kiwi (fruit) center. Cute. Unfortunately, no video and no wiki page (yet).

Christian presented a new way to represent issues within OpenNMS, a “heat map“. It works with both alarms and outages.

Jesse presented something that literally gave me goosebumps. Using our new integration with Newts, you can search for similar data within OpenNMS. So if there is say, a spike, you can search through all the other metrics to see if there are other data sources that spike at the same time.

David S. presented a new northbound interface for sending alarms to other systems via JMS. He used ActiveMQ as a proof of concept.

Ron created a couple of new features. The first was the ability to see polls as events, including how long each poll took (if available). He also added the ability to create a consistent color scheme across performance graphs.

Umberto created another real exciting feature – the ability to export in real time OpenNMS events to Elasticsearch. Since OpenNMS can handle thousands of events a second, sending them to a system built to analyze such data could be very useful. Umberto was sponsored to attend by the OpenNMS Foundation.

A second OpenNMS Foundation attendee was Marcel. He worked on improving data collection for Fortinet devices.

Another cool feature was Dustin’s custom data collection script tool. Sometimes OpenNMS gets criticized for not using SSH to collect data and perform montoring. The reason it doesn’t is rather simple: it’s a stupid idea. It usually requires that you set up keys with null passphrases, and often they connect as root. Despite the security issues, it is also a resource hog and can’t scale. We have always recommended using an extensible SNMP agent like Net-SNMP, but it can be some effort to set up.

Dustin’s feature allows you to put collection scripts in a special folder on the server, and OpenNMS will automatically collect the data. All you need to do then is to add a graph definition and you’re done.

Ronny discussed running OpenNMS in Vagrant and Docker containers. Neato.

DJ was frustrated in how long it can take to compile OpenNMS if tests are enabled. OpenNMS is heavily instrumented with software tests. These can be broken into “unit” tests and “integration” tests. Maven (the build system used by OpenNMS) can be configured to separate them. Since unit tests should be small and quick, they can be run every time with the integration tests only run for regression.

Finally, Seth presented the work he did this week which was focused on changes needed to support OpenNMS Minion. Minions are little, stand alone processes that can perform basic monitoring and data collection, which is then forwarded up to an OpenNMS Dominion instance.

While the last Dev-Jam always seems to be the best Dev-Jam, I think it was true this year. This work will go along way toward positioning OpenNMS for the coming Internet of Things, and as always it is amazing to see what brilliant people can do when given the opportunity to work together.

We ended the day at Surly Brewing Company. The beer was delicious and the company stimulating. I only got one non-blurry picture, and unfortunately my pants fell down when I stood up to take it.

Sorry.

I hope to see everyone at the OpenNMS Users Conference in September. I promise to keep my pants on.

It’s hard to believe that this year’s Dev Jam is half over. It seems to take forever to get here and then it is over too soon.

Lots of cool stuff going on. David Schlenk has written a Java Message Service (JMS) northbounder interface. While targeted at Apache ActiveMQ, it should work with any JMS provider, and it is another great tool to have for the OpenNMS platform.

Christian did a cool little hack using a Blink(1) USB light. If you have a Blink(1) you can plug it into your laptop and then run a little Java app. The app will connect to your OpenNMS instance and then the color will change based on the severity of alarms. Cool.

I also participated in my first GPG key signing. Jeff organized it to increase our “web of trust” and once I got into it, it was kind of fun.

He promised cake, and for once the cake wasn’t a lie:

The cake went well with dinner, which came from the always amazing Brasa:

Most of us ate out on the deck. This is the view looking out toward the Mississippi:

While we have held Dev-Jam in locations other than UMN, it has become a lot harder to do so since we get treated so well here. While most attendees have been to previous Dev-Jam events, we always have a few new people, and many of them end up staying off campus in a nearby hotel. There is something about staying in a dorm room that bothers them – perhaps it was a bad experience in college.

So I thought it would be a public service to actually show the rooms we get at Yudof Hall.

Each person at Dev-Jam gets their own room. While these rooms tend to hold at least two students when classes are in session, during the summer they are singles. You get a sink and a little kitchen with a microwave, two burner stove and refrigerator.

There is a single bed, desk, an armoire (closet) and a chest of drawers.

You even get to control the temperature in the room. I like mine on the cool side and the air conditioning works quite well.

Each room shares a bathroom with a toilet and a shower with one other person from Dev-Jam. While these rooms might be a little close with two people, they are perfect for one. Plus they are incredibly convenient since the event is held in the downstairs Club Room.

So, if you ever decide to come to Dev-Jam, don’t hesitate to stay on campus.