Author: Tarus

Expensify and Why I Hate the Cloud

Posted on by Tarus

Over the weekend I found out that Expensify, a service I use for my company, outsources a feature to Amazon’s Mechanical Turk service. Expensify handles the management of business expenses, which for a company like ours can be problematic as we do a lot of travel when deploying services. The issue is that the feature, the “smart scanning” of receipts, could potentially expose confidential data to third parties. As a user of Expensify, this bothers me.

Expensify touts “SmartScan” as:

As background, SmartScan is the patented, award-winning technology that underpins our “fire and forget” design for expense management. When you get a receipt, rather than stuffing it into your pocket to dread for later, just:

1. Take your phone out of your pocket
2. SmartScan the receipt
3. Put your phone back in your pocket

What they never told us is that if their “patented, award-winning technology” can’t read your receipt, they send it to the Mechanical Turk, which in turn presents it to a human being who will interpret the receipt manually. The thing is, we have no control over who will see that information, which could be confidential. For example, when I post a receipt for an airline ticket, it may include my record locater, ticket number and itinerary, all of which are sensitive.

This apparently never occurred to the folks at Expensify. Take this Quora answer from Ryan Schaffer, listed as Expensify Director of Marketing & Strategy:

Also, its worth mentioning, they don’t see anything that can personally identify you. They see a date, merchant, and amount. Receipts, by their very nature, are intended be thrown away and are explicitly non-sensitive. Anyone looking at a receipt isunable to tell if that receipt is from me, you, your neighbor, or someone on the other side of the world.

Wrong, wrong, wrong. It seems that Mr. Schaffer may limit his business expenses to the occasional coffee at Starbucks, but for the rest of us it is rarely that limited. For someone whose job is to perfect dealing with receipts, his view is pretty myopic.

For examples of what Expensify exposes, take a look at this tweet by Gary Pendergast.

Information Exposed by Expensify Tweet

It is also worth noting that it appears Expensify does its business on the Mechanical Turk as “Fluffy Cloud” instead of Expensify, which strikes me as a little disingenuous.

In a blog post this morning the company addressed this:

As you might imagine, doing this is easier said than done. Given the enormous scale and 24/7 nature of this task, we have agents positioned around the world to hand off this volume from timezone to timezone. Most of the US team is located in Ironwood, MI or Portland, OR (where we have offices and can train in person). Most of the international team is in Nepal or Honduras (where we work with a third-party provider to manage the on-site logistics). But regardless of the location, every single agent is bound by a confidentiality agreement, and subject to severe repercussions if that agreement is broken.

But if this were true, why are random people on Twitter announcing that they can see this data? Are they relying on the Amazon agreement with the people working as part of the Mechanical Turk? That doesn’t instill much confidence in me. But then in the same blog post they double down, and suggest that if you want extra security, you can just set up your own staff as part of the Mechanical Turk:

1. You hire a 24/7 team of human transcription agents.
         o For the fastest processing we suggest staffing three separate shifts — or daytime shifts in three different offices around the world. Otherwise your receipts might lag for many hours before getting processed.

2. They apply to Amazon Mechanical Turk for an account. Be aware that this is a surprisingly involved process, including:
         o The agent must sign up using their actual personal Amazon account. If your account doesn’t have an adequate history of purchases (each of which implies a successful credit card billing transaction and package delivery) or other activity, you will be rejected.
         o The agent must provide their full name, address, and bank account information for reimbursement. Amazon verifies this with a variety of techniques (eg, confirm that your IP is in the country you say you are, verify the bank account is owned by the name and address provided, full criminal background check), and if anything doesn’t add up, you will be rejected.
         o Rejection is final. It requires such an abundance of verifiable documentation (most notably being an active Amazon account with a long history) that you can’t just create a new account and try again.
         o There is no apparent appeals process. Accordingly, I would recommend confirming before hiring that the candidate can pass Amazon Mechanical Turk’s many strict controls because we have no ability to override their judgement.

3. You notify us of the “workerID” of each of your authorized agents.
         o Though you are not obligated to share your staff’s identity with us directly, your staff will still be obligated to follow the Expensify terms of services. Failure to comply with our terms will result in an appropriate response, starting with immediate banning by our automated systems, ranging up to our legal team subpoenaing you (or failing that, Amazon) for the identity of the agent to press charges directly.

4. We will create a “Qualification” for your “Human Intelligence Tasks” (HITs) that ensures only your agents will see your receipts.

5. Your staff will use the Amazon Mechanical Turk interface to discover and process your employee’s receipts.

That’s the solution? This is what passes for security at Expensify? Hire three shifts of employees all using verified personal Amazon accounts and then you can be sure your confidential data is kept confidential?

Wouldn’t it just be easier to create a small webapp that would present receipts to people in a company directly without going through the Mechanical Turk? Heck, why not just bounce it back out to you – it isn’t that great of a chore.

Plus, basically, if you don’t do this Expensify is saying they can’t keep your information secure.

This is what frustrates me the most about “the Cloud”. Everyone is in such a rush to deploy solutions that they just don’t think about security. Hey, it’s only receipts, right? Look what I was able to find out with just a discarded boarding pass – receipts can have much more information. And this is from a company that is supposed to be focused on dealing with expenses.

I demand two things from companies I trust with our information in the cloud: security and transparency. It looks like Expensify has neither.

I will be moving us away from Expensify. If you know of any decent solutions, let me know. Xpenditure looks pretty good, and since they are based in the EU perhaps they understand privacy a little better than they do in San Francisco.

2017 Cubaconf

Posted on by Tarus

I’ve just returned from Cubaconf in Havana, which was also my first visit to Cuba. It was a great trip and I’ve got enough material for at least four blog posts. Most of them won’t deal with free and open source software, so I’ll put them up on my personal blog and I’ll add links here when they are done..

Cubaconf is in its second year, and while I really wish they would have called it “Cuba Libré” (grin) it was a good conference.

There is a spectrum within the Free, Libré, Open Source Software (FLOSS) community, and this is often described by trying to separate the term “open source” from “free software”. If we define “open source” as any software with a license that meets requirements of the Open Source Definition (OSD) and “free software” as any software with a license that meets the requirements of the Four Freedoms, they are the same. You can map the ten requirements of the OSD onto the four requirements of free software.

Open Source is Free Software Chart

However, it can be useful to separate those who look at FLOSS as simply a development methodology from those who view it as a social movement. When companies like Microsoft and Facebook publish open source software, they are simply looking to gain value for their business that such sharing can create. It’s a development methodology. When people talk about free software, they tend to focus on the “help your neighbor” aspect of it, and this was more the focus of Cubaconf than simply creating new code.

The main thing I discovered on my visit was that Cubans face severe limitations on many things, but I’ve never met a people more determine to do as much as they can to make their situation better, and to do it with such passion. If I had to pick a theme for the conference, that would be it: passionate continuous improvement.

Cubaconf Registration

The three day conference had the following structure: Day One was a standard conference with keynotes and five tracks of presentations, Day Two had keynotes and more of a “barcamp” organization, and Day Three was set aside for workshops, as well as the obligatory video game tournament.

They did have the best lanyard sponsor I’ve seen at a technology conference:

Cubaconf Lanyard and Badge

I was in Cuba with my friend and coworker Alejandro, who used to live in Venezuela and is a fluent Spanish speaker, and Elizabeth K. Joseph, who promotes the open source aspects of Mesosphere. We shared a three bedroom “casa particular” in Old Havana, about a ten minute walk from the conference, which was held on the second floor (third floor if you are American) of the Colegio de San Gerónimo.

Everyone was together on the main room for the first keynote.

Cubaconf People in Room

While both English and Spanish were spoken at the conference, the presentations were overwhelmingly in Spanish, which was to be expected. I can get by in Spanish, but the first speaker, Ismael Olea, spoke fast even for the native speakers. At least I could understand most of the content in his slides.

Cubaconf Ismael Olea

Olea is from Spain and did a keynote on HackLab Almería. Almería is a province in the southeastern part of Spain, and with a population of around 700,000 people it is much smaller than provinces like Madrid (6.5 million) and Barcelona (5.5 million). As such, the region doesn’t get as much attention as the larger provinces, and so they goal of Hacklab Almería is to use technology at the “hyperlocal” level. They define themselves as a “collective of technological , social and creative experimentation” and FLOSS plays a large role in their mission.

After the keynote, we broke up into individual sessions. I went to one called “How to Make Money with Free Software” presented by Valessio Brito from Brazil. While he spoke mainly in Spanish, his slides were in Portuguese, but I was able to follow along. His presentation focused on how he used his knowledge of FLOSS to get consulting engagements around the world. This was pretty topical since in Cuba, as elsewhere, having strong software skills can be lucrative, and since a lot of proprietary software is either impossible to get or too expensive, having skills in open source software is a plus.

Cubaconf Valessio Brito

Also, I liked his shirt.

Our OpenNMS presentation was in the next time slot. I asked our hosts if they would like the presentation in English or Spanish, and when they said Spanish I asked Alejandro to give it. He did a great job, even though he had only a short time to understand the slides.

Cubaconf Alejandro Galue

The lunch break came next, and we walked a couple of blocks to the Casa de Africa, a museum dedicated to the African influence in Cuban culture.

Cubaconf Lunch Break

We ate sandwiches and talked out on the patio. This would be the location for lunch for all three days.

Cubaconf Maira Sutton

After lunch I watched a presentation by Maira Sutton called “Fighting Cyber Dystopia with Tech Solidarity and the Digital Commons” which is a long way to express the idea of using free software combined with working together to take back some of the power from large corporations. Her main example talked about the city of Austin, Texas, and its fight with Uber and Lyft. Austin wanted all ride sharing drivers to have to undergo a background check that included fingerprints. Sounds reasonable, but Uber and Lyft resisted, eventually leaving the city.

However, services like Uber and Lyft can be downright useful, so a number of startups filled in the gap, offering similar services that met the City’s fingerprint requirements. Instead of acquiescing to local laws, Uber and Lyft took their fight to the State legislature, which overturned Austin’s regulation.

Even though it is a sad ending, the example did demonstrate that combining technology and social action can result in solutions that can meet or exceed those provided by large commercial companies.

Cubaconf First Night's Event

For each night of the conference there was an event, and the one for Tuesday was held at a modern art gallery on the southern side of Old Havana. There was lots of food and drink, and I got exposed to a project called cuban.engineer. This is a group to promote technology careers within Cuba, and I had seen their shirts at the conference.

Cubaconf cuban.engineer shirts

In a lot of the world we take Internet access for granted. I can remember accessing the Internet from the night market in Siem Reap, Cambodia, on an open wi-fi connection. That doesn’t exist in Cuba. Cuba is one of the most disconnected countries in the world, which can make working with technology difficult. Access is controlled by an agency called ETESCA. To access the Internet you purchase a card which offers a certain number of hours of use, and then you have to locate an area with a wi-fi hotspot (usually near a park). The card has a number of digits for a username and a number of digits as a password, and once you get connected you hope you stay connected long enough to do what you need to do.

No one is exempt from this. Even in our apartment the owner would use one of these cards to enable access for the hotspot on the ground floor. So, if you are a technology business in Havana, your first job is to located your office near a hotspot, and then buy a bunch of these cards.

Thus you can imagine that sharing in a big part of the culture. People burn and swap CDs with software such as Ubuntu on them, and they tend to use Gitlab to make local mirrors of code repositories. While wi-fi equipment can be hard to come by, people have been able to set up their own, private wi-fi networks within cities like Havana to make sharing easier. There is no Internet access (I joked that it was Cuba’s “dark web“) but they can set up tools like Rocket.Chat to communicate and share.

Despite limitations in acquiring software, Microsoft Windows is still the most common operating system running on Cuban computers. An attempt was made to create a Cuba focused Linux-based distro called Nova. I was told that they even experimented with making it look as close to Windows 7 as possible, but people were still tied to using Windows. According to Wikipedia this distro is no more, which is a shame.

Cubaconf Mixæl Laufer

The second day started with the meter pegged at full on social justice, with a presentation by Dr. Mixæl S. Laufer, Ph.D., from Four Thieves Vinegar. They are a collective aiming to share information on how to create pharmaceuticals in places where they might not be available. If you live in the US than you probably heard of Martin Shkreli who as CEO of Turing Pharmaceuticals raised prices 5600%, and EpiPen maker Mylan who raised the price of this life saving device several hundred percent just because they could. Laufer showed how you could make your own EpiPen for around $30, among other things.

Now drug companies will say that they have to charge that amount to cover the costs of creating new drugs, but the EpiPen greed demonstrated that wasn’t true. Running health care as a “for profit” enterprise has always seemed inherently wrong due to the incentives being more toward making money versus keeping people healthy, but that is commentary for another time and place.

I had to leave after that presentation for something special. I make classic cocktails as a hobby, and one of our hosts asked me to speak to a school for bartenders (cantineros) on the great tradition of Cuba cocktails. It was a blast and I’ll write that up soon.

Cubaconf Wednesday Event

Wednesday night’s event was held, appropriately, at a bar in an area called Barber’s Alley. It was a fun gathering and I got a nice picture of some of our hosts.

Cubaconf Hosts

Left to right is Jessy, Pablo, pb, and Adalberto.

There was also a guy there who made pipes, specifically replicas of native American peace pipes, and one was passed around.

Cubaconf Peace Pipe

The third and final day was a series of workshops, but was started with a keynote from Ailin Febles, from the Uniōn de Informāticas de Cuba, a non-profit organization to bring together “all technicians, professionals and people related to information and communication technologies in a space that enables mutual support of the associates in the achievement of their professional, academic, scientific, cultural and personal objectives”.

Cubaconf Ailin Febles

Of course, a lot of their organization is driven by open source software.

Cubaconf Software used by UIC

I hope they switch to Nextcloud from Owncloud soon.

There was one morning workshop in English, ironically by a German named Christian Weilbach, on machine learning. I was interested in the topic since I keep hearing about it lately, and the fact that I would probably be able to understand it was a plus. To me machine learning is magic, and I wanted to dispel some of that magic.

Cubaconf Christian Weilbach

It worked. It turns out that machine learning is, to a large extent, what we used to call linear algebra. It just is able to work on much larger and more complex data sets. I’m still eager to play more with this technology, but it was nice to learn that it really isn’t all that new.

Cubaconf Old Car Taxi

After lunch we decided to spend our last afternoon exploring Havana a bit.

Cubaconf Brewery Event

The final evening event was in a brewery, and I enjoyed the beer. What I enjoyed more was the opportunity to talk with Inaury about race in Cuba. Cubans come in all shapes and sizes, from people with light skin, blond hair and blue eyes to people so dark they are almost blue, yet they all seem to interact and socialize with each other more so than any other place I’ve been. I plan to chat more about that in a blog post as well.

Overall I had a great time in Cuba. I love the fact that working in free software means I can make new friends in almost any country, and that even a place with limited resources can put on a great conference. If you get a chance to go to Cubaconf, you should take it.

2017 Ohio Linuxfest

Posted on by Tarus

The Ohio Linuxfest was one of the first open source conferences I ever attended, way back in 2010. I had heard about it from the local Linux Users Group, TriLUG, and we ended up renting a van and taking a couple of other open source geeks with us, including “Mr. IPv6” Kevin Otte.

It was a blast.

Ohio Linuxfest Banner

This year looks like another great show, with one of my favorite people, Karen Sandler, giving a keynote on Friday and yours truly will be giving the last keynote on Saturday.

If you are into free and open source software and are able to make it, I strongly encourage you to check out the conference. You’ll be glad you did.

2017 Australian Network Operators Group Conference

Posted on by Tarus

Back in June I was chatting with “mobius” about all things OpenNMS. He lives and works in Perth, Australia, and suggested that I do a presentation at AusNOG, the Australian Network Operators Group.

One of the things we struggle with at OpenNMS is figuring out how to make people aware it exists. My rather biased opinion is that it is awesome, but a lot of people have never heard of it. To help with that I used to attend a lot of free and open source conferences, but we’ve found out over the years that our user base tends to be more along the lines of large enterprises and network operators that might not be represented at such shows.

Imagine my surprise when I found out that there were a whole slew of NOGs, network operator groups, around the world. It seems to me that people who attend these conferences have a more immediate need for OpenNMS, and with that in mind I submitted a talk to AusNOG. I was very happy it was selected, not in the least because I would get to return to Australia for a third time.

AusNOG David Hughes

I wasn’t sure what to expect, but was pleasantly surprised. The event was extremely well organized and I really liked the format. Many conferences as they become successful respond by adding multiple tracks. This can be useful if the tracks are easy to delineate, but often you can get “track bloat” where the attendees get overwhelmed with choice and as a presenter you can end up with a nearly empty room if you are scheduled against a popular speaker. At AusNOG there is only one track of somewhat short, highly curated talks that results in a very informative conference without the stress of trying to determine the best set of talks to attend.

AusNOG Program

(Note: Visit the “programme” site and click on talk titles to download the presentations)

The venue was very nice as well. Held at the Langham Hotel, the conference took place in a ballroom that held the 200+ people with a lobby out front for socializing and a few sponsor booths. The program consisted of 90 minutes of presentations separated by a break. They alternated sets of three 30 minute talks with two 45 minutes talks. I found all of the presentations interesting, but I have to admit that I spent a lot of time looking up unfamiliar acronyms. As network operators Autonomous System (AS) numbers were thrown around in much the same way SNMP Private Enterprise Numbers are shared among network management geeks. Australia is also in the process of implementing a nationwide National Broadband Network (nbn™) to provide common infrastructure across the country, so of course that was the focus of a number of talks.

In the middle of each day we broke for lunch which was pretty amazing. The Langham restaurant had a sushi section, a section for Indian food, a large buffet of your standard meat and veg, and at least three dessert sections: one with “healthy” fruit and cheese, another focused on ice cream and a chocolate fountain, and a large case full of amazing pastries and other desserts. All with a view out over the Yarra river.

AusNOG Yarra River

I really liked the format of AugNOG and suggest we adopt it for the next OpenNMS conference. For those few talks that were either over my head or not really of interest, they were over pretty quickly, but I found myself enjoying most of them. I thought it was interesting that concepts we usually equate with the managing servers are being adopted on the network side. One talk discussed topics such as running switch software in containers, while another discussed using Ansible and Salt to manage the configuration of network gear.

AusNOG Runing Switch Software in Containers

I was happy to see that my talk wasn’t the only one that focused on open source. Back fifteen years ago getting large companies to adopt an open source solution was still in the evangelical stage, but now it is pretty much standard. Even Facebook presented a talk on their open source NetNORAD project for monitoring using a distributed system to measure latency and packet loss.

I did have a few favorite talks. In “The Future Is Up in the Sky” Jon Brewer discussed satellite Internet. As someone who suffered for years with a satellite network connection, it was interesting to learn what is being done in this area. I used a system with a satellite in geosynchronous orbit which, while it worked, ended up with latency on the order of a second in real-world use. It turns out that there are a number of solutions using shorter distances with satellites in low earth or high earth orbit. While they present their own challenges, it is still the most promising way to get network access to remote areas.

Another talk by Mark Nottingham discussed issues associated with the increased use of encrypted protocols and the challenges they create for network operators. While the civil libertarian in me applauds anything that makes it harder for surveillance to track users, as a network monitoring guy this can make it more difficult to track down the cause of network issues.

And this will become even more important as the network changes with the adoption of Internet of Things (IoT) devices. Another good talk discussed the issue of IoT security. Even today the main consensus is that you protect weakly secured devices with a firewall, but a number of new exploits leverage infected systems within the firewall for DDoS attacks.

AusNOG Internet of Things Vulnerabilities

I think my own talk went well, it was hard to squeeze a good introduction to OpenNMS into 30 minutes. I did manage it – 30 minutes on the nose – but it didn’t leave time for questions. As a speaker I really liked the feedback the conference provided in the form of a rather long report showing what the attendees thought of the talk, complete with cool graphs.

AusNOG Speaker Response

I really enjoyed this conference, both as an attendee and a speaker. While I hope to speak to more NOGs I would much rather encourage OpenNMS users who are happy with the project to submit real-world talks on how they use the platform to their local tech groups. I think it tells a much stronger story to have someone a little less biased than myself talking about OpenNMS, and plus you get to visit cool conferences like AusNOG.

Help Get OpenNMS Packaged by Bitnami

Posted on by Tarus

As someone who has used OpenNMS for, well, many years, I think it is a breeze to get installed. Simply add the repository to your server, install the package(s), run the installer and start it.

However, there are a number of new users who still have issues getting it installed. This is not a problem limited just to OpenNMS but can be a problem across a number of open source projects.

Enter Bitnami. Bitnami is a project to package applications to make them easier to install: natively, in the cloud, in a container or as a virtual machine. Ronny pointed out that OpenNMS is listed in their “wishlist” section, and if we can get enough votes, perhaps they will add it to their stack.

Bitnami also happens to have a great team, lead in part by the ever amazing Erica Brescia. As I write this we have less that 50 votes, with the current leaders being over 1200, so there is a long way to go. I’d appreciate your support, and once you vote you get a second chance to vote again via the socials.

Bitnami OpenNMS Wishlist

Thanks, and thanks to Bitnami for the opportunity.

2017 Australian Network Operators Group (AusNOG) Conference

Posted on by Tarus

I am excited to be returning to Australia for the third time next week. This trip is to speak at the Australian Network Operators Group (AusNOG) annual conference in Melbourne.

AusNOG Promo Graphic

I can’t believe I’ve gone for so long and not known about Network Operator Groups (NOGs). There are quite a few of them and I think they would be a perfect audience to introduce to the OpenNMS Project. One of our users on the OpenNMS chat server is from Perth and he made me aware of the conference, and I was humbled and delighted to have my presentation accepted.

At OpenNMS we strive very hard to separate the project (.org) from the commercial entity that supports the project (.com) and this presentation will be strictly focused on the project. It’s a wonderful thing about OpenNMS: if it meets your needs, cool. If not, also cool. I just want more people to be aware of open source options, especially in the carrier and enterprise space.

And it looks like open source is definitely making inroads at AusNOG. The talk before mine is about Ansible and Salt. There is another talk on using open source to build a version of NetNorad, and another one on open source for big data analytics. The one after mine is about modern network monitoring, so I hope I tick at least a few boxes on his list.

I hope to see you there (although it looks like it is sold out) but let me know if you are in the area and perhaps I can at least say “hi”.

2017 Dev-Jam – Days 4 and 5

Posted on by Tarus

Apologies to my three readers for the two-plus week delay in writing this up. I know you’ve been waiting for this post with more anticipation than Season 7 of Game of Thrones, but things have been crazy busy in OpenNMS-land of late.

As you know, this year’s Dev-Jam was held at Concordia University in Montréal. For most of the week we had access to a nice, large space which included air conditioners (the Grey Nuns building does not have central air), but due to a scheduling conflict we had to move out for the day on Thursday.

We moved to the basement cafeteria, which worked out rather well.

Dev-Jam: Grey Nuns Cafeteria

There were no A/C units but with the windows open there was a good cross breeze and it was comfortable. By the fourth day of Dev-Jam people are pretty much settled into a routine, so the day was mostly spent working to finish up various projects.

On Friday we moved back upstairs. The last full day of Dev-Jam is always bittersweet for me, as I love the “show and tell” part where people share what they have been working on, but I hate the fact that it is over for another year. We also forgot to bring the equipment we usually use for video capture (it was left back in Minnesota) so while the sessions were recorded, they haven’t been posted to Youtube yet.

Dev-Jam: Jesse White presents on Minion

Jesse kicked off the presentations talking about work he was doing to assign specific monitoring tasks to particular Minions.

Dev-Jam: Markus presents on Doughnut Graphs

Markus followed that with his work on adding “doughnut” graphs to the user interface. These resemble the graphs available with Compass™, our mobile app. That is supposed to be a green doughnut and not a grey one but the projector didn’t render it very well.

Dev-Jam: Christian presents on IFTTT

As OpenNMS wants to be the monitoring platform of choice for the Internet of Everything, Christian did some work on integrating it with “If This, Then That” (IFTTT)

Dev-Jam: Ronny presents on ASCIIBinder

Ronny is our “documentation czar” who led the effort to create the most awesome docs.opennms.org site. He explored using ASCIIBinder to manage our growing collection of documentation.

Dev-Jam: Seth presents on ReST

I work with some amazing people, and years ago they saw the potential in adding ReST functionality to OpenNMS. It was a great decision and makes OpenNMS extremely flexible when integrating with other systems. Seth presented some of the work he is doing to extend that feature.

Dev-Jam: Alejandro presents on Drools

OpenNMS has a couple of ways to correlate alarms. The basic method is using the “vacuumd” configuration and SQL, but a stronger (although more complex) method is to used the Drools business intelligence engine. Alejandro presented some work he is doing to move some of the legacy vacuumd tasks to Drools.

Dev-Jam: Jeff presents on CLA Assistant

Contributor License Agreements are a controversial topic in open source, but we use one for OpenNMS. The main reason is to ask any contributor to certify that they have the right to contribute the code. It may seem trivial, but not only does signing such an agreement make the person think about it, it does give the project some cover in case of a dispute.

We currently manage our own CLAs, but the website CLA Assistant aims to make it easier. Jeff presented on what he found out about the service, and we may be migrating to it in the future.

Dev-Jam: Antonio presents on Enhanced Linkd

And last but not least, Antonio talked about the work he is doing on Enhanced Linkd. This is the process that figures out Layer 2 connections between devices. It is non-trivial as vendors seems to relate this information in different ways, and we really appreciate the time he has put into that part of the project.

It was great seeing everyone again, and it was also cool to hold the conference in a new location (well, cool once it was over, it added a lot of stress to my life). I’m eager to start planning next year’s conference.

2017 Dev-Jam – Day 3

Posted on by Tarus

By Day 3 we’ve settled into a rhythm, so I don’t have much to report from the actual OpenNMS side of things. Personally, I spent way too much time trying to figure out why Twitter is blocking links to this blog. It’s been ad-free for over a decade, yet Twitter thinks it is spam.

We believe it was because we were using dlvr.it to post things I write here to the OpenNMS Twitter feed and thus it got flagged as automation (which is, apparently, bad). I can understand it, but my complaint is that there is no clear process for getting it resolved. I think I’ve submitted the proper request and I’ve even tried to back-channel some help through friends of friends, but I think I’m just going to focus on posting on Google Plus from here on out, unless I need to complain. (grin)

So the gang worked while I bitched at Twitter. Oh, I do have a picture of a Canadian food product: ketchup flavored potato chips, modeled by Jeff.

Dev-Jam: Ketchup Potato Chips - photo credit Jessica

I can report on what we did in the evening. Usually Dev-Jam involves seeing a Major League Baseball game, specifically the Minnesota Twins. However, the only MLB team ever in Montréal, the Expos, played their last season here a year before Dev-Jam started (2004).

However, Montréal does host an international fireworks competition, so we got tickets. It’s held at La Ronde, which is an amusement park in the Six Flags chain. La Ronde is located on the northern end of St. Helen’s Island (Île Sainte-Hélène). While the park itself wasn’t very crowded, it turns out that Metallica fans, including our own Alejandro, were descending on the southern part of the island for a concert.

To get there from Grey Nuns, we took the Metro. The Metro station on St. Helen’s Island is near a museum called The Biosphere, complete with a geodesic dome created by Buckminster Fuller. Might be worth a return visit.

Dev-Jam: Montréal Biosphere - photo credit Mike

We took a free shuttle to get to the park, and our tickets also granted us admission. Out of habit I had taken my grandfather’s pocket knife, not realizing they would have metal detectors, but they had a cool system where I could drop it off and pick it up later (so I didn’t have to hide it in the bushes).

As soon as we got into the park, the sky opened up and poured on us. Some of us found shelter under building overhangs, some on rides and some just got wet. One of the rides was the carousel, where Ben, Seth and Cynthia seemed to have fun.

Dev-Jam: La Ronde Carousel - photo credit Mike

Luckily, the rain didn’t last too long. Since it was National Hot Dog Day, we decided to find some. As a fan of the show Silicon Valley, I suggested we try out the “Not Hotdog” app. Seems to work.

Dev-Jam: Not Hotdog - screenshots Mike

Afterward there was just enough time for a ride on Le Vampire.

Dev-Jam: Le Vampire Rollercoaster - photo credit Chris

Jessica is in the back of this shot, with Ronny and Jesse in the front.

Then it was time to see the show. There are three seating sections: Bronze, Silver and Gold. Gold had free beer but we had been told that the Silver section (Argent) had the best views. I wasn’t disappointed. And this being Quebec, there had to be clowns.

Dev-Jam: Fireworks Clowns - photo credit Ben

In section 307 we were on an elevated platform looking out over a lake. The fireworks were launched from the other side of the lake and synchronized with music. As this is an international competition, the presenter for our show was Germany. They played the German national anthem and introduced the performers.

Dev-Jam: Fireworks Introductions - photo credit Ben

The show … it was amazing.

And when I say it was amazing, I mean it was like I had never seen fireworks before, and I’ve seen them at places like EPCOT at Disney World. Since we were pretty close to the launch site, it was like they took up your whole field of vision. Plus, I never knew there were so many different types. They used the whole area, from high overhead down to the lake itself (at one point in time it was if the water was ablaze in green fire). Shells would explode overhead in a burst of color, followed by a second color and when those faded there would be a ring left that slowly faded away.

Dev-Jam: Fireworks Above - photo credit Ben

Sometimes the sky would just be a fountain of gold, and at other times the action would be at ground level. At times I felt they were telling a story. Not exactly one I understood, such as the red fountain thingies seem to be fighting the shooty yellow thingies, but it still evoked an emotional response.

Dev-Jam: Fireworks Down Low - photo credit Ben

The emotional high point for me was when they played Pachelbel’s Canon. I know it is cliché but the “Canon in D” is one of the most beautiful things ever made, and to see it illustrated in fire was simply breathtaking.

Speaking of things to take your breath away, soon after the show started the air was filled with smoke and ash from the fireworks. While it really didn’t detract from the show, people with health issues related to breathing should think carefully about attending.

Dev-Jam: The Bridge at Night - photo credit Ben

The show was 30 minutes long, and by the time we headed out to leave so did 40,000 Metallica fans. This was further complicated by the bridge to the island being closed so people could watch the fireworks. Rather than waiting for buses to arrive once the bridge was reopened, we decided to walk. There was a great view from the top, and the bridge itself was lit in green.

Dev-Jam: View from Bridge - photo credit Ben

We got back to the dorm around midnight, having had a great time. I hope I get a chance to see the fireworks show again, perhaps at next year’s Dev-Jam.

2017 Dev-Jam – Day 2

Posted on by Tarus

Dev-Jam was fully underway by Tuesday morning, starting with another Canadian tradition, Tim Hortons.

Dev-Jam: Tim Hortons Box

Lots of great discussions were going on. Ronny demonstrated Project Atlas, more formally known as the GraphML Topology Provider. This allows you to use GraphML to create topologies within the OpenNMS user interface.

Dev-Jam: Ronny Doing a Demo of Atlas

There is also a topology.xml file on the OpenNMS Forge github repository that can be used as an example.

Jesse gave us a demonstration of Project Helm. This is a Grafana plugin that let’s you combine fault and performance data from multiple instances of OpenNMS on one dashboard.

Dev-Jam: Jesse Doing a Demo of Helm

While it is just in alpha, the goal is to let users manage alarms directly from the dashboard, including acknowledging them, adding “sticky” and journal notes, etc. We have been working for years now on making a robust ReST interface for OpenNMS and it is really paying off by allowing us to create features like this. Since all of the communication between Grafana and the OpenNMS system (or systems) is via ReST, there is no need to store and manage data locally.

Dev-Jam: Helm Screenshot

If you want to play with Helm, you should be running the latest Horizon 21 snapshot.

We have a person named Roberto attending Dev-Jam for the first time and I was eager to find out why he was interested in OpenNMS, so I spent some time talking with him. His company deploys underwater fiber-optic cable. Their customers used to be large telecommunications providers, but now they deal mainly with very large Internet companies, and those companies are requesting a higher level of monitoring information. It was one of those “very large Internet companies” that suggested they use OpenNMS, and it was interesting to learn about the challenges of running and managing undersea fiber.

Only a small part of the cable contains the fiber as most of it consists of a thick protective sleeve. The sleeve has to be thicker near shore since there is a greater chance of damage from things like ships’ anchors. Also, electrical current flows through the sleeve which attracts sharks, who then proceed to bite the cable. Here’s a video:

I’m eager to see how they end up using OpenNMS.

As I mentioned before, we are staying in the Grey Nuns Residence at Concordia University. According to Wikipedia there is a crypt in the basement where nearly 300 bodies are buried, most of them nuns who had lived at the Grey Nuns Motherhouse. It is off limits to visit, but I wanted to see if I could at least find the entrance.

Our conference is being held in a large room called E104, and most of the rooms of the people attending are also on the east side of the residence. My room, however, is on the west side and to get there I have to walk about 200 meters (it is a big place). You go out of E104, down to the basement and along a very long corridor before heading up several floors.

Along this corridor you will see a nondescript door,

Dev-Jam: Door to Grey Nuns Crypt

and if you peek through the little round window you can see into the crypt.

Dev-Jam: Image of Grey Nuns Crypt

The graves are marked with plain wooden crosses, and the one nearest the door died in 1885, although there are certain to be much older graves in the crypt. Apparently there was a project to move the bodies out of Grey Nuns but the government balked due to the fact that some of the people buried there died of infectious diseases (the history of the Grey Nuns [pdf] confirms that several nuns died of the Spanish Flu of 1918).

It is a pretty solemn place and in stark contrast to the rest of the dormitory.

Speaking of things definitely not solemn, for dinner we all headed to a Japanese restaurant nearby called Kinka Izakaya. Izakaya means a pub, and the menu consists of lots of small plates, kind of like Japanese tapas.

Dev-Jam: Dinner at Kinka Izakaya

The place met a number of criteria: good food, can seat 24 people and close to the dorm. We also had to try a “Sake Bomb” in which a small amount of sake is suspended over a glass of beer. You then drop the sake into the beer and drink. Yes, there is video:

Good times.

2017 Dev-Jam – Day 1

Posted on by Tarus

Dev-Jam is an unstructured conference. Our goal is to simply put a bunch of incredibly smart people in a room and see what happens. That said, we do officially start and end the conference. On Monday morning we get together to make introductions and to talk about projects that we want to pursue during the week. This allows people with similar interests to work together if they want. On Friday we have presentations on what got accomplished.

Dev-Jam: People Around a Table

I usually start off the week, and then turn it over to Jesse White (our CTO and GM of The OpenNMS Group Canada).

Dev-Jam: Jesse White in an MC Frontalot shirt

I thought it was cool that he was wearing an MC Frontalot T-shirt as we commissioned him to produce a free software song that we released on Independence Day (July 4th).

Another cool thing about OpenNMS is that we try to work as transparently as possible. While a lot of projects allow public access to their git repository, I believe OpenNMS is the only one that has a repository for every branch that automatically builds packages as commits are made (the list can be found at http://yum.opennms.org/repofiles/ but be patient as there are so many it can take a minute or so to load). We also publish a weekly newsletter called “This Week in OpenNMS” (or TWiO). This week Ben posted some ideas bouncing around this year’s Dev-Jam which include:

  • updating packaging (yum/Debian) infrastructure including better support of upstream PostgreSQL packages
  • improved wifi link support in Enlinkd and topology
  • improving the opennms.com and opennms.org web sites
  • simplifying collection of OCAs
  • integrating DigitalOcean and Xen requisition tools
  • Spark chat integration
  • ReST infrastructure improvements
  • discussing how to improve Docker image generation
  • grafana dashboard for UPS data
  • northbound interfaces for Drools (scriptd-like interface for alarms)
  • structured data monitor (using the XML collector infrastructure)
  • Cisco ACI integration
  • OSGi deployment of ReST services
  • donut charts on the front page
  • rewrite the node list page
  • migrate documentation to AsciiBinder
  • trigger IFTTT events when alarms change
  • porting the Go version of the minion to a new platform

Dev-Jam: Laptop and Can of Cheerwine

I should note that Ben is also a fan of that North Carolina export, Cheerwine.

Dev-Jam: People Meeting to Discuss the Website

Jessica, our graphic designer, pulled together a meeting to discuss our web presence. We recently revamped the opennms.com website and we are looking to determine improvements needed for the opennms.org website.

Speaking of Jessica, she also designed our Dev-Jam shirts.

Dev-Jam: Front of Dev-Jam Shirt

The front is meant to represent summer camp.

Dev-Jam: Back of Dev-Jam Shirt

The top symbol is for Montréal, the bottom is Ulf the OpenNMS mascot, the tag to the left represents coding and the right image is for fireworks (we are attending the competition on Wednesday).

It’s not all work. Recently I read about a restaurant very close to Concordia (where we are holding the conference) that offered free meals to people who need them. I wanted to support that, so for lunch a group of us went to Marché Ferdous, which had been written up on sites such as CNN, the BBC and Huffington Post.

Dev-Jam: Marché Ferdous Entrance

It’s a small shop a couple of blocks away from our dorm, and I got the falafel platter.

Dev-Jam: Marché Ferdous Falafel Platter

My meal was about CAD$10 so I paid with a $20 bill and told them to keep the change. Always gotta pay it forward, yo. (grin)

The falafel was just okay (I’m spoiled as I get to eat Angie’s falafel on a regular basis – some of the best falafel on the planet and I should know as I’ve had it pretty much all over the planet) but the sides were excellent. Everyone else got meat and really enjoyed it.

After lunch we took a side trip to a SAQ store to check out the spirit selection. Later that evening there was a tasting …

Dev-Jam: Monday Whiskey Selection

… which probably had something to do with the decision to screen Strange Brew, eh?

Dev-Jam: Strange Brew on Screen