Portlandia

Last week I was delighted to return to Portland, Oregon, for the annual OSCON conference. While I had a fun time at OSCON, I was only there for Wednesday and this post is about Tuesday. I’ll talk about the conference itself next. So this post contains little OpenNMS content but might be interesting to those of my three readers who like travel.

I got to PDX around noon and took the MAX into the city. My plan was to drop some stuff off at the hotel and then head to the conference, but as usual my life was overtaken by events. By the time I dealt with my various e-mails and other obligations, it was fairly late in the afternoon so I opted for a nap before dinner instead of trekking over to the Convention Center.

Dinner included me, our OpenNMS guru extraordinaire Ken who was up from Salem, and Greg. Greg used to be an OpenNMS client before he changed jobs to work for a non-profit, but like all OpenNMS users he is super intelligent and amazing to be around. I never miss a chance to spend some time with him.

Ken showed up in his cool, new BMW diesel wagon and drove us across the river. A friend of mine who is also a chef recommended we try a place called Pok Pok and Greg seconded the recommendation. He was worried that it might be crowded, but it being a Tuesday night we figured we’d take our chances. It was also a little earlier than I usually eat dinner, but even then our wait for a table was an hour.

In the meantime we went across the street to the Whiskey Soda Lounge. Greg told us that when Pok Pok started to get popular, people would migrate to nearby bars to wait for their table. Sensing a business opportunity, when a building close to the restaurant became available, they opened up the Lounge.

While we didn’t order much food, we did have a couple of drinks waiting for our table. I really enjoyed the Tamarind Whiskey Sour, which is something of a signature drink. They also had some decent drink specials. The only food we got was a bowl of peanuts, but in keeping with the Thai style cuisine of Pok Pok, the peanuts came mixed in with mild chilis. While the Lounge was nothing out of the ordinary, it was nice to unwind before dinner and catch up.

Our table was ready in an hour as promised, and the wait staff in the Lounge were the ones to tell us about it (which I thought was pretty convenient). Now Pok Pok is one of those places that seems to have grown organically, and our table was in a section that was basically outdoors with a covered roof. Considering how nice the evening was, I preferred our table to the ones downstairs in the main restaurant which struck me as a little claustrophobic.

The food was good and unusual. Greg and I are pretty much “pescatarians” so we stuck with fish. We had catfish (Cha Ca La Vong)

as well as whole tilapia, which was a special.

Afterward, for dessert Greg suggested that we go to an ice cream shop called Salt and Straw.

Now I am an ice cream nut. I make it at home. I go to classes to learn how to make better ice cream. And my favorite ice cream is Jeni’s from Ohio which is simply fantastic.

How do you describe something that is nearly an order of magnitude better than fantastic?

I’m not sure if it is the creaminess, the wonderful flavor combinations or just high quality ingredients, but this stuff is good.

Greg told me that the long line was actually pretty short for this place but I still I felt bad when I got up to the front and wanted to try every single flavor as the line got longer behind me. I limited myself to five, and received a heaping tablespoon of each one to taste.

The first was the signature Sea Salt with Caramel. Wonderful. I’ve just been introduced to salted caramel as a flavor and I’m quite fond of it.

Since I couldn’t just stop on the first one, I also tried their Double Fold Vanilla and Chocolate Gooey Brownie. Both were superlative.

The fourth flavor was Lavender and Honey. When we walked in another customer was walking out with a big purple scoop, so I just had to try it and this was almost my choice but I had to try one last flavor: Strawberry Honey Balsamic with Black Pepper.

As the youths like to say: OMG.

I love strawberries and this flavor combination just caught my eye. One of the issues with making great ice cream is to limit water. Water forms ice crystals which ruin the texture on the tongue. This makes working with fruit difficult due to its high water content. Usually when I make strawberry ice cream, I roast the fruit to get rid of some of the water, and then I puree it and mix it in with the cream. You don’t want chunks because they screw up the texture when they freeze.

Salt and Straw gets around this by not only pureeing and mixing in the fruit, they have made a type of jam that they swirl into the ice cream. So you get amazing amounts of fruit flavor without sacrificing the texture. This was a softer ice cream than the others but it was so, so, good.

Both Ken and Greg thought the ice cream as good as well. Greg pulled a “native” by also purchasing a pint to go. You can skip the line and head straight to the to go freezer, but I wanted to get back in line to try some more flavors so I doubt I could do that.

Anyway, it was a great “soft landing” for my trip. I also got a few laughs when talking to the locals. I’d tell them, in my deepest southern drawl, that I knew all about Portland from watching that documentary on the city: Portlandia.

What makes that show funny is the deeper truths it parodies, but that’s one of the things I love about that town.

♫ It’s Hard Out Here for a (Free Software) Pimp ♫

In thinking about a title for this particular screed, I almost went with “Papa’s Got a Brand New Phone” but that didn’t really encompass what I was after as much as a play on the Oscar winning “Best Original Song” by the Three 6 Posse.

When I first got involved in free software, I thought it was too good to be true. I thought “free” implied “no work” but I was confusing free (gratis) with free (libre).

Sometimes freedom takes work.

It takes effort and no small commitment to run as much free software as possible, and no where is that more evident than when it comes to choosing hardware.

I used to be a big Apple fanboy, and thus my personal technology decisions were easy: buy the newest shiny from Apple. When I decided to divorce myself from them, it took awhile to adjust to the fact that, quite frequently, the new shiny is not the best choice for a free software advocate.

But I’ve been stymied time and time again. When looking for a new laptop, I bought the latest Lenevo X1 Carbon and ended up sending it back. It was just too new to support my operating system of choice, whereas my old, second generation Dell XPS 13 “Sputnik” runs Linux Mint Debian Edition (LMDE) just fine.

So I tried a new tact.

When I was in the market for a new phone, I figured the best bet was to work backwards.

I had been using a Samsung Galaxy S3 running Cyanogenmod. However, right after I upgraded the baseband to run Kit Kat, the phone would constantly and randomly reboot. I tried everything I knew of to fix it and tried out just about every major ROM there was but it would still crash. Only by running Jelly Bean could I mitigate the issue somewhat. Then instead of bouncing every hour or so, it would only reboot once or twice a day.

Now I play a game called Ingress … a lot. It is a heavy user of the display, the CPU, the network and the GPS. While these reboots might have been acceptable to a casual user, they were killing me. While I may have somehow corrupted my S3, it was probably due to some other hardware problem, so I decided to get a new phone.

One of the pluses about putting in the time to use free software is quite frequently you learn how things work. I would never have even known about baseband versions, bootloaders, recovery, etc. if I hadn’t played with my phone. I also get a lot of options, such as which ROM to run. In all my research I decided that my philosophy matches up best with the team behind OmniROM.

OmniROM doesn’t have as many options as, say, AOKP, but they are dedicated to keeping it as open as possible and I admire that. Plus they have a pretty decent OpenDelta update application that makes staying on the latest release pretty simple.

Once I decided that I wanted to run OmniROM, I just worked backwards to pick out a phone.

Here’s where I had to make a choice about freedom.

What I loved about my S3 was that it had a replaceable battery and a microSD slot. Some days I’m a heavy user of my phone and even the best phones can’t last the day on a single charge. The microSD slot made it easy to transfer data from my phone to my computer as well as easily and cheaply expanding the available memory.

Not many phones have these two features. In fact, the only modern phones I could find were both from Samsung: the S4 and the S5.

The S5 is not supported by OmniROM, so my choice was simple: get the S4. I ordered an unlocked S4 from Amazon and got ready to enjoy the new-ish shiny.

It was not to be.

While the description on Amazon said that it was “unlocked” it turns out that Samsung has decided to block third party bootloaders, even on the S4, with an update issued last November, so it is impossible to replace their default operating system with a free one. While there are some ways to “dual boot” the phone, this was unacceptable to me, so I sent it back with the reason “item did not match web site description”. Just being carrier unlocked is not enough to merit the term “unlocked”.

In looking over the remaining options, I ended up settling on last year’s HTC One (m7). And I do mean settle: the One has no microSD slot nor does it have a replaceable battery. But these are things I can work around in the pursuit of freedom. I got a microSD to microUSB connector and an external battery pack that can keep my phone running for days. It also has a somewhat lo-rez camera at 4 megapixels, but it seems to take pictures just fine.

You do have to jump through an extra hoop in order to unlock the bootloader, but HTC made it pretty simple. You just have to log in to their developer site and post a code and they’ll send you back a file to run to unlock your particular phone. Not as easy as, say, a Nexus phone, but it isn’t too much extra work.

Now I have the latest Kit Kat running flawlessly on the phone. I’m able to remove the Google search bar, which in my case just takes up space, and I can modify the number of icons displayed per page.

It’s pretty awesome.

Is the HTC One a perfect phone, especially for playing Ingress? No – it is not perfect. But it is pretty darn good. At the Gettysburg anomaly it held up all day with zero reboots, whereas other people were reporting them with usually stable phones such as the Nexus 5. Note that if I didn’t have any other considerations I would have gotten a Nexus phone, but since I play Ingress with my spouse and she has one I wanted another brand in order to diversify the radio technology. In some places her phone gets signal where mine does not, and vice versa, and thus we can tether if needed.

I like to vote with my wallet and I buy products from companies that support freedom. I don’t understand why Samsung felt the need to lock down their devices. In part I think it is Apple-envy, but they just lost out to those of us who want to truly own their hardware. I’m not sure if it is enough to affect the bottom line, but it has soured me on Samsung products as a whole and I do buy a lot of technology.

So, remember that freedom takes work, but it’s worth it in the end.

Oh Nos! My Wireless Stopped Working!

I just had something a little scary happen, so I thought I’d share it in case anyone else hits this problem.

I’m in Portland for OSCON and suddenly the wireless networking on my laptop stopped working. The wireless status showed as “off” but it wouldn’t turn on. I’m running Linux Mint Debian Edition (LMDE) and no interfaces were showing up.

Now, one thing I like about open source is I always tend to learn something when trying to solve a problem. A quick search on my phone introduced me to the “rfkill” command:

# rfkill list
0: phy0: Wireless LAN
	Soft blocked: no
	Hard blocked: yes

For some reason, the interface was “Hard blocked”. I then figured out what must have happened.

I was trying to bring up a shell to diagnose another issue. On Linux this tends to be ALT+CTL+Fx where the function key chosen is the virtual terminal you want (i.e. F1 for the first one, F2 for the second, etc.). On my normal keyboard, which is an old Apple keyboard, the function keys default to softkeys and you have to hold down the Fn key to actually trigger F1, F2, etc.

This is not the case with my laptop, so when I hit Fn+F2 it turned on “airplane mode”. This was causing the hard block.

I hit it again:

# rfkill list
0: phy0: Wireless LAN
	Soft blocked: yes
	Hard blocked: no
1: hci0: Bluetooth
	Soft blocked: no
	Hard blocked: no

And then turned off the soft block:

orcrist interfaces.d # rfkill unblock 0

And it fixed my issue:

orcrist interfaces.d # rfkill list
0: phy0: Wireless LAN
	Soft blocked: no
	Hard blocked: no
1: hci0: Bluetooth
	Soft blocked: no
	Hard blocked: no

It would have really sucked to be on the road and have some serious software issue to repair with no network access, so I was extremely relieved to figure this out.

30 Years

I missed noting it, but 30 years ago on Saturday, June 9th, 1984 I graduated in the third graduating class of the North Carolina School of Science and Mathematics. It was an amazing two years and I’m happy to say that I’m still in touch with many of the people I met there.

The speaker at my graduation was Ron McNair, the first African-American astronaut. Earlier that year, in February, I got to see him launched into space on Space Shuttle mission STS-41-B. Unfortunately, he was one of the seven people killed in the Challenger disaster a couple of year later.

While most people, I would assume, don’t remember the speech given at their high school graduation, I remember him talking about daring to be “a flea weight in a heavy weight world”. I took those words to heart and it is one of the reasons OpenNMS is able to challenge products from the likes of HP and IBM.

I should note that on Sunday, June 10th, 1984, I started working as a machine operator in a plastics injection molding plant in Asheboro, NC. In retrospect, it wouldn’t have hurt me to take a day off but I guess that just how we open source geeks roll.

(grin)

2014 Dev Jam – Day 6

Friday was pretty much the “Dev Jam Results Show”. Mike suggested that the various teams get together and present their work. He recorded it live from his iPhone using Ustream and the raw (and I mean raw) video is available for your viewing pleasure.

The first stream covered the following topics.

Ben, Ron and Matt R. – AngularJS based webUI

That OpenNMS could use a new user interface is a given, and the decision was made to base it on a technology called AngularJS. The first demo shows off some of the work that was done to build a framework for the new GUI.

This includes a “plugin” architecture that will make it easy to add functionality to the system as well as to embed existing code that, while not written in Angular, is still useful. Ron even managed to get KSC reports to run under the new code, complete with dynamic updates.

Work that still remains to be done include formal authentication (currently the new GUI just gets a session cookie from the old one) as well as a greater granularity for ReST permissions, as now normal users get a lot of data and admin users get all the data. This would be very useful for things like multi-tenancy.

Alejandro – Requistion Manager

When we wrote the provisioner, we knew we had something special as no other management system seems to take discovery as seriously as OpenNMS. As more and more people find novel ways of using this system, we realized that the user interface could use some improvement. In this section Alejandro demonstrates the changes he has made to the interface for creating and managing requisitions, also built on Angular.

Craig Gallen – High Frequency Trader GUI

Those of you that follow OpenNMS in the news might have seen a press release a couple of weeks ago from a financial trading services company called TMX Atrium Networks. Dr. Gallen, our man in the UK, worked with them to build an interface for monitoring latency across the network.

Matt and Eric – newts

Yesterday I talked about the New Time Series database that we are building as the data storage backend for OpenNMS. I lifted a lot of that from this portion of the demonstrations. The ability to have an incredibly fast and highly scalable data store is key for our goal of making OpenNMS the de facto network management platform of choice.

The second video stream features a talk about “snee-po”

Seth – SNMnepO

SNMnepO, or OpenNMS spelled backwards, is a project to create a distributed data collector with horizontal scalability. The idea is to add data collection to our remote poller, and Seth’s demonstration shows data collection being performed with the collectd process disabled.

Coupled with the newts data storage backend, this new distributed collector will insure that OpenNMS can scale to meet any data collection needs in the future.

The final stream focuses on work being done by our German team.

Christian – Outage Timeline

Christian demonstrates the new outage timeline that I talked about earlier in the week.

Dustin – RRDtool export via ReST

Dustin shows a new feature that exposes collected data from the RRD files via ReST. This can allow for another integration point where collected data can easily be used by other applications.

Ronny – PRIS

The last demo was done by Ronny (presenting work that was also done by Dustin) on the Provisioning Integration Server (PRIS). As mentioned above, the ability to tightly integrate OpenNMS with provisioning systems is a key feature of the platform. Originally done to integrate with OCS Inventory NG, the system has been extended to allow for integration with pretty much any system.

Considering that these demonstrations were pretty much ad hoc, I was delighted to see how much was accomplished in just a week. It is one of the main reasons I look forward to Dev Jam every year.

We celebrated that evening with a trip to Republic.

Let’s just say the evening went a little downhill from there, but I did manage to make it back to the dorm.

2014 Dev Jam – Day 5

Thursday turned out to be picture day. We had some people leaving a little early so we decided to get our group picture done in the morning. Recently the school added a new Goldy Gopher statue near our dorm, so it was the logical place for a photo.

It also worked out that Goldy would fit in a 3XL OpenNMS shirt (grin)

I even took the opportunity for one of them there “selfie” thingies:

Don’t expect to see many more, but I was told it would be “ironic” (grin).

There was some real work done as well. We are getting much closer to a 1.0 release of newts (http://www.newts.io), the NEW Time Series database built on Cassandra. The speed is pretty amazing, with sustained writes of 50K+ data points per second.

For testing we’ve been using some weather data that contains 1.2 billion data points, but even at 50K per second it takes six hours to import.

Note that this was done on Matt’s laptop and one Cassandra node. On server hardware it should be much faster, and Matt and Eric have worked very hard to make it linearly scalable: two nodes are twice as fast, four nodes are four times as fast, etc.

The whole Internet of Things paradigm requires the ability to manage massive amounts of time series data and we are getting close to making it a reality.

I am also dealing with the reality that I ate way too much pizza in the last 24 hours. Thanks to Chris Rodman and the good people at Papa John’s Pizza, we had a pizza feast:

(burp)

Order of the Blue Polo Update

It has been awhile since we had an entry into the Order of the Blue Polo, so I thought it would be cool to blog about it.

While the Order of the Green Polo (OGP) is the governing body of the OpenNMS Project, we wanted to find a way to recognize users who didn’t quite have the time necessary to dedicate to the project for OGP membership. It also solves a problem for us: we have lots of amazing users of OpenNMS, but we can’t always talk about them.

Membership in the Order of the Blue Polo is pretty straightforward. Simply send us an e-mail on why you like OpenNMS, preferably with a quick list of the number of devices/interfaces/services you are monitoring. If we can publish it on the wiki along with your name and your company’s name, we’ll send you a limited edition blue OpenNMS polo. So please, no gmail.com or yahoo.com e-mails – we really need to be able to verify your company.

This helps us because like attracts like, and perhaps someone will read about how you are using OpenNMS and decide that it fits in with their needs.

The latest entry into the Order comes from Paul Cole, a contractor and Environment Canada. He writes:

I am a contractor working currently for Environment Canada (under Shared services Canada).

It was a beautiful thing to be given the OpenNMS project in order to map out and bring together teams to monitor and work on the very large LAN. A multitude of tools and scripts were used custom to each area, and we are now moving towards unification.

The visibility and baseline abilities of OpenNMS are fantastic, and the new topo-mapping/geo-mapping features are looking fantastic come version 1.14!

Our network size is larger than the current scope of nodes we are testing with, but OpenNMS is managing it pretty smoothly and seamlessly.

I have released some of the scripts I wrote to contribute back to the community that are based on version 1.10-1.12 and hope they help more people realise the power and scalability of the product.

Currently our OpenNMS build is monitoring over 15k nodes and 20k interfaces and 25k or more services (exact numbers can be extracted but it grows every quarter), on an 8 core server with 16GB of RAM using the discovery method.

It is keyed to discover every IP and node it can, and monitor switches/UPSs and routers and select key devices for management, to send email alerts to the appropriate regional teams when a device is down or a specific threshold or alert is received.

I can’t of course send out network diagrams , but I can send a screenshot of the geo-map to give an idea of how it goes.

Anarchy OpenNMS in the UK

OpenNMS has a strong presence in both Europe and the UK, and much of the UK effort is driven by Dr. Craig Gallen.

He has created a new website and newsletter aimed at OpenNMS users in the United Kingdom and Ireland (but, of course, it is open to anyone).

The new website can be found at opennms.co.uk and I think it is pretty spiffy (“spiffy” is a proper English word, correct?). There is also an occasional newletter list focusing on OpenNMS events in the region, so if you are interested in such things please register.

The first big push to raise awareness of OpenNMS as well as provide training is a series of OpenNMS workshops to be held around the area. In Craigs words:

Don’t just expect to be lectured to. This will be a participative event. These workshops will stretch your understanding of Operational Support systems and help you to begin thinking through how you can adapt OpenNMS to address some of the key problems in Network and Service Management.

London – Monday 30 June 2014

Location: University of London Union, Malet Street, London, WC1E 7HY

Birmingham – Tuesday 1 July 2014

Location: IET Aston Court, 80 Cambridge Street, Birmingham, B1 2NP

Rochdale (near Manchester) – Wednesday 2 July 2014

Location: Zen Interent Ltd. Sandbrook Park, Sandbrook Way, Rochdale, OL11 1RY

Glasgow – Friday 4 July 2014

Location: IET Glasgow: Teacher Building, 14 St Enoch Square, Glasgow, G1 4DB

There is a cost associated with the workshops, but there are a number of discounts available. There is an early bird discount of 10% if you book before 13 June, and if you are a current commercial support customer or a non-profit there is a further reduction in cost. Also, sending more than one person creates even more discounts.

So if you are a non-profit, buy a commercial support contract and then book a whole bunch of people before 13 June and you’ll be saving money hand over fist (grin).

Visit the Registration Page for more details.

This is a wondeful way to get up to speed on OpenNMS and I appreciate the effort Craig put into making these workshops available.

2014 Dev Jam – Day 4

First let me interrupt this blog post with a special announcement. A rather onerous security bug was discovered in OpenNMS that would allow any authenticated user to access pretty much any file on the system.

We felt it was bad enough to actually create a fix in the 1.10 branch as well as in the current stable, 1.12, so please consider upgrading at the earliest possibility.

Hat tip to Martin Laercher for reporting it.

Wednesday marked the halfway point in the week, and everyone seems to be in a good groove. With everyone able to work together in person, a lot of nifty things are getting done, including an upgrade to the latest version of Drools Expert.

The integration of OpenNMS with Drools allows for very powerful alarm correlation, and by migrating to Drools 6.0.1 it just got more powerful.

Wednesday also marked the day of the Twins game. For the past two years we’ve taken everyone to the Twins ballpark to watch a major league baseball game. It’s a beatiful place for baseball:

although they usually stuck us in far right field. Also, for the last two games the Twins played the Royals, and lost both times.

This year they put us in far left field:

and the Twins faced the Brewers. Since Milwaukee is close to Minneapolis there were a lot of Brewers fans in the stands, but the home team pulled it out for the win.

We got our name on the big board, too, which was cool, and Jeff was quick enough to catch a picture.

2014 Dev Jam – Day 3

By Tuesday things are pretty much underway. People have divided up into teams and are busy working on various improvements to OpenNMS.

All of this is managed by Bamboo, our build and test system. We have a dashboard view of the status of each branch: green means good and red is bad.

Last year DJ was working on a Raspberry Pi controlled stoplight, and it is now complete and tied into Bamboo. Again, green means good and red means bad, but it also pulses when building.

Tuesday evening we decided to head to Mall of America for some blatant consumerism (and dinner). I hired a school bus to drive us there, and while a “short bus” might have been more appropriate for this crowd, we had too many people to fit.

As part of the evening’s festivities, we attended a Star Trek exhibition at the Mall.

It contained a number of costumes and props from the various Star Trek television series and movies. It wasn’t too amazing but it was fun, and folks seemed to enjoy it.