Author: Tarus

Server Room Nightmares

Posted on by Tarus

I’m interested in any server room nightmares people would like to share.

Here’s one of mine.

We are in the process of moving offices from Pittsboro, NC down the road to Apex. Unfortunately, we are having some issues getting Spectrum Enterprise to complete the fiber installation at the new place, so while we are out of our old building the lack of network access in the new building means we have a bunch of servers in the old location.

Today while I was working in the new office and mooching of our kind neighbor’s wi-fi, I got several notices that links had failed.

linkDown event list

These were some workstations that we use for training, but when they are not in use we use them as part of our continuous improvement Bamboo farm. I immediately hopped on our Mattermost IT channel and asked if anyone was rebooting or otherwise messing with the machines, and when the answer was “no” I started to investigate.

One suggestion was that the air conditioning may have failed and those machines shut down from overheating. It has happened in the past, but it was both rather cool today and other machines that are more sensitive to such things were still running. I checked it out anyway using our AKCP probe.

temperature graph

The temperature had increased a bit, but it wasn’t anything that should have caused problems (it was caused by the server room door being left open).

Being 30 minutes away, I decided to text my friend Donnie, who is technically gifted as well as working in our old location, and he went to investigate.

For some reason, those three machines had been disconnected from the switch.

Now just for this situation we have an Arlo camera installed in the server room, so using the time stamp on the linkDown traps I found the following video.

Note the slightly balding guy in the red shirt in the lower left corner of the video. He is busy unplugging our devices.

Why? I have no idea. These people represent the IT people for the new tenant, and I assume they had legitimate reasons for being in the server room but messing with our equipment was not one of them.

Seriously, in over 30 years of working with computers, I’ve never heard of anyone going into someone’s house, office, server room or data center and just start unplugging cables. I still have not heard an explanation, but the landlord has had a discussion with the new tenant and it shouldn’t be happening again. It is one reason the important stuff is in that locked half-rack seen in the upper left corner of the video, and the really important stuff is hosted elsewhere.

I am curious – I’m certain this pales compared to other stories out there. Do you have any whoppers to share?

New Meridian® Releases Available

Posted on by Tarus

Just a quick note to point out that new Meridian releases are now available: 2015.1.5 and 2016.1.5

For those who aren’t aware, Meridian is a subscription-based version of OpenNMS built to complement Horizon, the cutting edge release. You can think of it as Meridian is our Red Hat Enterprise Linux to Horizon’s Fedora. There is one major Meridian release per year and each major release is supported for three years.

Before the Meridian/Horizon split it was taking us 18 months or so to do a new major release of OpenNMS. Now we do three to four Horizon major releases a year.

About half of our revenue comes from support contracts and so we had to be extra careful when doing a release, and even with that many of our customers were reluctant to upgrade because the process could be involved. This was bad for two main reasons: often they wouldn’t get bug fixes which meant an increase in support tickets, and more importantly they might miss security updates.

Updates to Meridian, within a major release, are dead simple. This is the process I used yesterday to upgrade our production instance of OpenNMS.

First, I made a backup of the /opt/opennms/etc and /opt/opennms/jetty-webapps/opennms directories. The first is out of habit since configuration files shouldn’t change between point releases, but the second is to preserve any customizations made to the webUI. I modify the main OpenNMS page to include a “weather widget” and that customization gets removed on upgrades. Most users won’t have an issue but just in case I like having a backup.

Next, I stop OpenNMS and run yum install opennms which will download and install the new release. The final step is to run /opt/opennms/bin/install -dis to insure the database is up to date.

And that’s it. In my case, I copy the index.jsp from my backup to restore the weather information, but otherwise you just restart OpenNMS. The process takes minutes and is basically as fast as your Internet connection.

If you have a Meridian subscription, be sure to upgrade as soon as you are able, and if you don’t, what are you waiting for? (grin)

OpenNMS Team Wins 5000€ Prize at TM Forum {open}:hack

Posted on by Tarus

A group of four students from Southampton Solent University, mentored by Dr. Craig Gallen, used OpenNMS to win the top prize at the TeleManagement Forum {open}:hack competition at the TM Forum Live conference in Nice, France.

{open}:hack Winners

Now, a little background is in order. Dr. Gallen founded Entimoss, our OpenNMS partner in the UK and Ireland. He got involved with OpenNMS over a decade ago when he was working on his doctoral thesis entitled “Improving the Practice of Operations Support Systems in the Telecommunications Industry using Open Source”.

Most of his work was focused on a business solution framework called NGOSS (now Frameworx) developed by the TM Forum for creating next generation OSS/BSS software and systems. Now the TM Forum is the world’s leading trade organization for telecommunications providers and at the time was not very friendly toward open source. He demonstrated how an open source platform like OpenNMS could be used to integrate with and tie together these different interfaces to build a reference implementation for part of the framework. Open source was a new concept for the industry, and we were branded the “open source pirates” at first. But Craig persisted, and in 2011 he was awarded the TM Forum’s Outstanding Contributor Award.

In addition to his persistence and ability to deal with large organizations, Craig is also a great teacher. When the TM Forum introduced its {open}:hack program, he wanted to get involved and he found several interested students at Southampton Solent University.

The goals of {open}:hack are:

  1. Accelerate industry deployment of Forum Open APIs, metamodels and architecture across the industry
  2. Validate existing APIs and provide feedback for future iterations to technical collaboration teams
  3. Create IoT/Smart City & NFV/SDN solutions leveraging the Forum Open APIs
  4. Accelerate the incubation of new digital business opportunities in the areas of 5G Network Services & IoT/Smart City
  5. Create extensions to Forum Open APIs to be shared with industry

Participants were given access to APIs from the TM Forum, Huawei, Salesforce and Vodafone, which included things like data from drones, and tasked with creating something beneficial. Their project was called “Port-o-matic” which created an application for accessing services at shipping ports, as well as measuring environmental factors such as pollution. This was especially relevant to them since Southampton is the UK’s number one cruise port and second largest container port (the Titanic set sail from there).

{open}:hack architecture

Their solution leveraged the power of the OpenNMS platform to tie all of these APIs together and then to provide aggregated data to their web application. It can scale to almost any size using the new OpenNMS “Minion” feature which can distribute data collection and monitoring out to the edges of a network, offloading the need to have all of the functionality in a central location and positioning OpenNMS for the Internet of Things (IoT).

The hardest thing to get across to people new to OpenNMS is that it is a platform and not strictly an application. The learning curve can be steep and it is hard to see its value straight out of the box. I love the fact that solutions like the “Port-o-matic” demonstrate the power of OpenNMS.

It is also interesting to note that the second place prize went to a team from Red Hat. For an organization like the TM Forum that was wary of open source to demonstrate such a change of heart is encouraging, and I credit Dr. Gallen with a lot of that advancement.

{open}:hack Group Photo

So congratulations to Joe Appleton, Jergus Lejko, Michael Sievenpiper and Marcin Wisniewski, the winners of this latest {open}:hack competition, and I look forward to seeing more great things from you in the future.

2017 Red Hat Summit

Posted on by Tarus

I had never been to a Red Hat Summit before this year. We are exploring running OpenNMS on OpenShift and so Jesse, David and I decided to head to Boston to see what all the fuss was about.

RHSummit - Airline Sign

I noticed a couple of things are different about visiting Boston in spring versus winter. First of all, the weather was quite nice, and second, Boston can be freakin’ expensive.

And Red Hat spared no expense on this conference. This is the premiere event for companies in the Red Hat ecosystem and they obviously wanted to make an impression. I’m an “old guy” and I can remember going to huge shows put on by HP and IBM and this was on par. It took place at the Boston Convention and Exhibition Center (BCEC) which takes up about a half a million square feet. Red Hat used all of it.

RHSummit - Convention Center Sign

Nothing quite demonstrated the size of this conference than the main auditorium. The centerpiece was a huge screen for the presentation flanked by two smaller screens to show the speaker. That was needed since the place was so big you could barely see the person talking.

RHSummit - Main Auditorium Screen

In addition to the general sessions, there were a large number of talks on pretty much anything related to Red Hat products, philosophy and partners. As a major player in “the cloud” there was a lot of emphasis on OpenShift and OpenStack, but the whole range of offerings was covered from Fedora and CentOS to JBoss and Gluster.

As with most tech conferences, there was an expo floor. This one was dominated by the color red.

RHSummit - Expo Floor

I spent a lot of time wandering around talking with people. Over the years a large number of my friends have been hired by Red Hat, and as I’ve curtailed my participation in a lot of the big Linux conferences, it was nice to see them again. I ran into Brian Proffitt and Ruth Suehle near the center of the expo:

RHSummit - Brian Proffitt and Ruth Suehle

It was also nice to run into the Latvian army. The Zabbix crew had a booth and it was cool to see Alexei and Alex again, although it was ironic that I missed them on my trip to Riga (they were actually driving north to Tallinn when I was heading south).

RHSummit - Zabbix Booth

Zabbix, like OpenNMS, is 100% open source and thus not only do we get along, I quite like them and look forward to chatting about the joys and challenges about running an open source business when we meet.

Speaking of meeting, I also got to meet Brian Stinson of the CentOS project.

RHSummit - Brian Stinson from CentOS

We swapped some stories and recounted the strange and funny time when Jerry Taylor, the City Manager of Tuttle, Oklahoma, claimed the CentOS project had hacked his city’s website. Has it been eleven years? Wow.

As part of the conference, Red Hat provided lunch. It was always a pretty hectic time since the show was packed and nothing demonstrated this more than trying to serve lunch to all those people.

RHSummit - Lunch Crowd

As far as conference lunches go, it was above average, but I did find it funny that they only served water to drink (usually there are cans of soda, etc.) I overhead one Red Hat employee say to another, you know, we can afford that gigantic screen but all we get is water?

On Wednesday night, Red Hat purchased a ton of tickets to the Red Sox game at Fenway Park. While I can’t find a reference to actual conference attendance figures, I heard the number 5000 being batted around which was a significant portion of the ballpark (it holds a little over 37,000). They gave us all red baseball caps and you could definitely see them in the crowd.

RHSummit - Fenway Park

For our annual developers conference, Dev-Jam, we have about one-one hundredth the amount of people to see the Twins play, but we also get better seats. (grin)

It was my first time at the historic Fenway Park, and the fans were almost more fun to watch than the game. I also enjoy trying to explain the game of baseball to people from outside the country, and this was made more interesting by some bad blood between the Sox and the Orioles that resulted in the ejection of the Orioles’ pitcher for hitting a batter.

Fenway is relatively close to Cambridge, so I took the opportunity to visit a friend of mine who is a professor. I decided to walk to Harvard Square along the river, where the rowing teams were practicing.

RHSummit - Rowing

Now whenever I see a movie featuring Ivy League students on the water, I’ll know where that was shot.

It was also nice to be able to spend some time with David and Jesse. While I work with David almost daily, we’re so busy that it is hard to find time to talk strategy and plan for the future of OpenNMS. Jesse, our CTO, moved back to Canada after the birth of his son to be closer to family, and it was also nice to have time to spend with him. Walking to dinner one night David took this picture

RHSummit - River and Bridge

which turned out so much better on his iPhone 6S than my Nexus 6P.

I often say that Red Hat, as a company, doesn’t get the credit it deserves since it is headquartered in North Carolina and not Silicon Valley. Our companies share a similar philosophy of taking care of customers, creating great open source software and producing steady growth, versus, say, chasing unicorns. It was wonderful to see that work demonstrated in such a large and professional conference, and I hope next year I’ll get to speak (although I doubt it will be on the big stage).

Fifteen Years

Posted on by Tarus

On Sunday my mother celebrated her 75th birthday.

Although a happy occasion, why is this relevant to an open source blog? Well, it was soon after her 60th birthday in 2002 that I started my first company around OpenNMS.

I did not start OpenNMS, it began in the summer of 1999, with the first code posted on Sourceforge in March of 2000 by a company called Oculan. I started working with Oculan in September of 2001, and in May of 2002 they decided to stop contributing to OpenNMS. I saw the potential, so I asked Steve Giles, the founder and CEO, if I could have the OpenNMS project. He looked at his watch and said if I was off his payroll by Friday, he’d give me the domain names, a couple of servers, and he would sprinkle water on me and I would be the new OpenNMS maintainer.

That was actually the easy part. Explaining to my wife that I had quit my job and started a company “selling free software” was a bit harder.

sortova.com from archive.org circa May 2002

And thus Sortova Consulting Group was born. It was named after my farm. When Andrea and I decided we wanted to have a farm, we first bought raw land. In driving out from Raleigh to work on it we would pass this little farm with a barn, some cows, etc., and on the mailbox was a sign reading “Almosta Farm”. I joked that if that was “almost a farm” then what we had was just “sort of a farm”. Later, when we bought the place where we still live, the name Sortova Farm stuck.

We pronounce it “Sore-toe-va”. Only one customer ever pulled me aside and asked if it really meant “sort of a” consulting group. He laughed when I confirmed that it did.

Considering that I didn’t have any prior business experience, Java experience, or even real Internet access at my home, it is amazing that OpenNMS survived to this day. It is a wonder what you can accomplish with pure stubbornness.

Now my one true superpower is my ability to get the most fantastic people on the planet to work with me. The first group of those came from the OpenNMS community. When I was running Sortova it was the gang that later became the Order of the Green Polo that kept me going, mainly through mailing lists and IRC. In September of 2004 my good friend and business partner David Hustace and I founded the OpenNMS Group, and that corporation is still going strong. In 2009 we mortgaged our houses to buy the copyright to the Oculan OpenNMS code and thus brought all of it back under one organization, and two of the original OpenNMS team at Oculan now work for OpenNMS.

When I visit Silicon Valley I often get to meet some brilliant people, but the joy of this can be offset by the pervasive attitude of focusing on technology simply to make money. I know of a number of personally successful people who built companies, sold them, and then those products vanished into obscurity. Remember VA Linux? Their stock rose over 700% on the first day of trading, but where are they now? Did they ever deliver on their promises to the stockholders?

I want to build with OpenNMS something that will last well beyond my involvement with the project. I’ve gotten it to the point where I am not longer expressly required to make it thrive, but I am still working on its legacy. We want it to be nothing less than the de facto standard for monitoring everything, which is a high bar.

Note that I still would like to make a lot of money, but that isn’t the core driving force of the business. Our mission statement is “Help Customers – Have Fun – Make Money” in that order. If you have happy customers and happy employees, the money will come.

Fifteen years ago I made a leap of faith, in both myself, my family and my friends. I’m extremely happy I did.

Privacy and Trash

Posted on by Tarus

Meet Sam. Sam is in his early twenties and grew up in Lake Mills, Wisconsin. He graduated from the University of Wisconsin in Madison in 2012. He is currently on vacation in Athens, Greece, with his girlfriend Sara. They managed to find an amazing deal on American Airlines from Minneapolis to Athens for $200 for the both of them, but with taxes and fees that ballooned up to nearly $850.

I have a copy of Sam’s resume, his Gmail address and his phone number. I know how long he’ll be gone and what seats they will be sitting in on their return. In fact, I know a lot more about Sam and Sara (Facebook and its ilk are ubiquitous) but I’m a little uncomfortable revealing as much as I have, so I’ll stop.

It is all because of this:

Sam Boarding Pass

With all the focus recently on the security of devices like those that make up the Internet of Things, what is often forgotten is that traditional paper has huge security issues in today’s connected world.

Airlines still insist on printing first and last names along with record locater codes on boarding passes. That is often all that is required to access a particular reservation. From there you can get information such as e-mail addresses and phone numbers.

This reminds me of when credit cards first came out and to use one the merchant would take an actual imprint of the card on carbon copy paper. Since that included the shopper’s name, complete card number and expiration date, it became easy for thieves to steal this information. At least now almost all receipts include, at most, the last for digits of the card (in case you were wondering, Sam used a Mastercard ending in 3286).

The genesis of this post arose from a more malicious reason. I fly a lot and over the years commercial air travel (which is the only air travel I can afford) has become less of a special occasion and more like taking a bad bus trip. People use the “seat back pocket” as their personal trash can, to the point that I almost never use it myself, even when I get upgraded to first class. Nasty. On this trip, the duration from when the last person got off the inbound plane until we started boarding our flight was less than ten minutes, so trust me when I say little was cleaned between flights.

I don’t blame the airlines. Consumers have spoken, and what they want is cheap airfare, so it is up to us to be respectful of our fellow passengers.

Anyway, when I see folks like Sam leave information like this as trash, I am so tempted to do things like reassign his seat to one in the middle next to the lavatory (it’s an 11 hour flight), or to cancel his flight completely. Lucky for him I believe in karma, and I just can’t bring myself to do it.

The basics of security involve two things: something you have and something you know. We need to apply this to everything that needs to be secure. I get so frustrated with systems in the United States, such as the new “chip” cards being used for credit and debit. Introduced a decade ago in Europe, their systems use “chip and PIN” – something you have, your card, and something you know, your PIN. In the US we are moving to “chip and signature” – something you have, your card, and something anyone can fake in a heartbeat, your signature.

(sigh)

This is especially touchy since two summers ago my spouse had her purse stolen. We immediately canceled and closed all of the accounts, but they were still able to get over $2000 out of our checking account. They used a paper check from another theft and then they cashed it at the bank using her ID. The bank forgot the “something you know” part of security even though they were quite aware that our account had been compromised and the account number changed. Only after the fact did they offer to “flag” transactions on our account for extra scrutiny, and now neither of us carries paper checks, although thieves could probably guess our bank from our ATM debit cards (we did get our money back from the bank).

So be careful. Buy a good shredder. If you need to dispose of paper when traveling, tear it into tiny bits and drop it in the nastiest trash can you can find … and not in the seat back pocket.

LinkedIn

Posted on by Tarus

I’m at Red Hat Summit in Boston this week so expect a longer post on the conference later, but I wanted to mention that I’ve reopened a LinkedIn account after an absence of several years. You can find me here:

https://www.linkedin.com/in/tarusbalog

I left the network due to how they were handling privacy issues. I’m still not 100% happy with it now, but I think I can control how much information I share and I do have a need that I think the service can provide.

I was walking in Boston yesterday and I saw a sign for Harvard Medical School. They used to use OpenNMS and I really enjoyed working with the guys who worked there. Most of them have moved on, so I was curious to know where they were and if they were still in the city. It dawned on me that LinkedIn would have helped in this situation.

I don’t like a number of changes that have been made to the site, such as the inability to feature external links (such as to this blog which will remain one of my main ways to communicate) but it may be just my inability to navigate the website. OpenNMS is also on LinkedIn, and it looks like you can “follow” the company as well:

https://www.linkedin.com/company/the-opennms-group

Anyway, let’s give this a go. See you in the toobz.

How Version 2.0 Killed Android Wear

Posted on by Tarus

I am the happy owner of an LG Urbane smartwatch. Unfortunately, I just upgraded to Android Wear 2.0 and now I can’t use it.

Andrea Wear 2.0 Upgrade

Luckily for me, my smartwatch is not “mission critical”. If I leave it at home by mistake, I don’t turn around to go back to get it. The main thing I use it for is notifications. I like the fact that if it is with me, it will automatically mute my phone and then vibrate when I have a notice. A quick glance at my wrist will tell me if I need to deal with it right this moment, or if it can wait.

The second thing I use it for is to do simple voice searches or to set reminders and timers. Outside of that there are a few apps I use and I like the fact that it tracks my steps, but overall I don’t use a ton of features.

When the notice popped up that I could upgrade, I blindly went ahead and did it. In retrospect, that was stupid, but I often get in trouble rushing out to install the “new shiny”. The upgrade seemed to go fine, and I didn’t think that much about it until lunch.

One of the things I do before heading out to lunch is check the temperature to see if I need a jacket. So I did the usual wrist flick to “wake” the watch and said “Ok Google” to get to the voice prompt.

Nothing happened.

Hrm, I did some research and apparently with 2.0 you have to press the button on the side of the watch to get to the Google prompt. I think this is a huge step backward, because now I have to involve both hands, and I find it ironic that with Android Wear 1.5 I I had to sit through a demo of one-handed gestures over and over again (I often have to re-pair my watch due to reloading software on my phone) and now they’ve thrown “do everything with one hand” out the window.

Anyway, I pressed the button which then brought up the Google Assistant setup screen on my phone. With 2.0 if you want to use voice searches, etc., you must use Google Assistant and you have to give Google access to all of your contacts, calendars etc.

(sigh)

I work hard to “sandbox” my Google activity from the rest of my digital life. It’s not that I think they are evil, it’s just that I don’t want anyone to have that much information on me, well, other than me. I kind of despair for free and open source software solutions in the consumer space. Everyone seems to be rushing to adopt these “always on” digital assistants with absolutely no regard to privacy, and this is causing vendors to lock down their ecosystems more and more. While open source is definitely winning on the server side, I don’t think the outlook has ever been so grim on the consumer side.

There were some upsides with 2.0, such as improvements to the look and feel, but I also found that I didn’t care for the new notification system (I seemed to miss a lot of them – perhaps I needed to change a configuration). But the requirement for Google Assistant was a deal breaker.

I thought about going back to 1.5, which I liked, but I can’t seem to find a factory image. In trying to locate one, I discovered that TWRP does have a version for bass (the codename for the LG Urbane) and I should have installed that and made a backup before upgrading. I contacted LG and they told me it was impossible to downgrade. That’s a load of crap because I could easily sideload the old version if they made it available, but then I’d have to deal with constant upgrade reminders and the few apps I do use would probably stop support for 1.5 to focus on 2.0.

It just isn’t worth it.

I know at least one of my three readers is thinking I should just cave and learn to embrace the Google, but I can’t bring myself to do it. I am eagerly awaiting open source alternatives like Asteriod OS (which just isn’t ready for daily use) and Mycroft (which is supposed to be shipping units this month) but I really don’t think I’ll miss my Urbane enough to spend the time on it.

I plan to sell my Urbane on eBay and I’ve gone back to my previous “dumb” watch (a nice little Frederique Constant I bought on a flight from Dubai to London). It’s kind of a shame since I enjoyed using it, but to be honest I’m not going to miss it all that much.

The Importance of Contributor Agreements

Posted on by Tarus

One thing that puzzles me is the resistance within the open source community to contributor agreements. This was brought into focus today when I read that the OpenSSL Project wants to migrate to the Apache 2.0 license from the current project specific OpenSSL license.

In order to do that they need permission from all of the nearly 400 contributors of the project over the last 20+ years, and contacting them will be a huge undertaking. If one person refuses to agree, then they will either have to abandon the effort, or locate that person’s contribution and either remove or replace it.

Many years ago we found out that a company was using OpenNMS in violation of our license. When our lawyer approached them about it, they claimed that they were only using those parts of the code for which we didn’t hold copyright. At that time, early versions of OpenNMS were still copyright Oculan, the company that started the project, and not OpenNMS. Since Oculan wasn’t around anymore it took us awhile to track down the intellectual property, but in the end David and I were able to mortgage our houses to purchase that copyright so that now the project can control all of the code and defend it from license abuse in the future.

But the question arose about what to do moving forward, specifically how should we deal with community contributions? In the past companies like MySQL required all contributors to sign a document with phrases like “You hereby irrevocably assign, transfer, and convey to MySQL all right, title and interest in and to the Contribution” which seemed a little harsh.

I posed this question to the Order of the Green Polo, the de facto project administrators, and DJ Gregor suggested we adopt the Sun Contributor Agreement that we now call the OpenNMS Contributor Agreement, or OCA. This was a straightforward document that asked two things.

First, you attest that you have the right to contribute the code. This is more important than you know, because it helps remove liability from the project should the contribution turn out to be encumbered in some way, such at the author writing it as part of their job and thus it is actually the property of the employer. We allow both individuals and companies to sign the OCA.

Second, you assign copyright to OpenNMS while retaining copyright yourself. This introduces the concept of “dual copyright”. Now some critics will say that this concept hasn’t been tested in court, but there is a long history of authors sharing copyright. Considering that Oracle maintained the agreement in the form of the Oracle Contributor Agreement, it appears that their lawyers were satisfied.

I claim responsibility for the license under which these Contributor Agreements are published: the Creative Commons Attribution-Share Alike License. When DJ suggested the Sun Contributor Agreement I noticed that there wasn’t any license on the agreement itself. I didn’t want to just copy it and change “Sun” to “OpenNMS”, so I contacted Brian Aker who had just moved to Sun with the MySQL acquisition and asked him about it. Soon thereafter the Agreement was updated with the license and we adopted our version of it.

Once we adopted the OCA, I was tasked with tracking down anyone who had ever contributed to OpenNMS outside of the company or Oculan and asking them to sign it. They all did, but I can tell you that I had a hard time tracking down a number of them (people move, e-mails change). I don’t envy OpenSSL at all.

I hope this story illustrates the importance of some sort of Contributor Agreement for open source projects. They don’t have to be evil, and in the end getting your copyright and licensing issues completely sorted out will make managing them in the future so much easier.

Electronic Devices and CPB

Posted on by Tarus

With the change in administration in the United States, Customs and Border Protection (CBP) have modified their behavior to include actions with which I don’t agree. These include forcing a US citizen to unlock his mobile device, even though it was a work device and contained sensitive information. I set out to come up with how I will deal with this situation should it arise in the future.

TL;DR My plan is as follows: before I enter the United States, I will generate a long, random password and set that as the encryption password for my laptop and my handy. I will then ssh into an old iMac I have on my desk, store the password in a file, and then shut the computer down. At that point I will not be able to access the information on my device until I return to the office and power on the system.

UPDATE: The EFF has published a detailed guide to help understand your rights at the border.

First off, let me say that until recently I’ve always respected CPB. They have a tough job and everyone I’ve ever met while returning from my travels has been efficient, competent and friendly.

But after the recent “Muslim Ban” fiasco I’ve come to realize that my experience is not universal. I think one of the main problems is this idea that the Constitution stops at the CBP desk, and until you are past it you really aren’t “in America” and thus the Constitution doesn’t apply.

I don’t agree with this interpretation, but it can probably be traced to the actions taken by the US government after 9/11 and the creation of the prison at Guantanamo Bay.

Prior to that, when “bad hombres” were captured by the US government, they fell into one of two categories: criminals or prisoners of war. How each class was treated was fairly well defined. Criminals were processed according to the rule of law, and the treatment of POW’s was covered under the various Geneva Conventions.

The US government decided that those two classifications were inconvenient, and so they ventured into the murky waters of “enemy combatant” and Guantanamo. Their logic goes that since Guantanamo isn’t in the US, US law doesn’t apply, and since these people aren’t members of a foreign country’s military force with which we are at war, then they aren’t POWs. So, the US gets to make up its own rules about how these people are treated.

This is dangerous for a number of reasons. Since nothing is really codified about the treatment and rights of the detainees at Guantanamo, the rules are arbitrary. Also, this opens the door for other countries such as Russia to do similar things without fear of international repercussions. The US has survived for so long because things like this are not supposed to happen, yet here we are.

This thought now extends to the border. Even though a US citizen is being questioned by another US citizen, in the role of a representative of the US government on US soil, somehow the rules of the Constitution are suspended. It’s arbitrary and I don’t buy it. The Constitution codifies a right to privacy in the Fourth Amendment, and it doesn’t go away when entering the country. And it definitely extends to mobile devices, which in today’s world are probably the most personal item people own.

So how can people like me, with almost no political power, resist this threat to our freedom?

I’ve always done little things, like opting out of millimeter wave scans at airports and getting a pat down instead (I’m not shy). If everyone did this the whole system would collapse, and they would find better ways of dealing with security than the security theater we have now. Seriously, if the Israelis don’t use it, it ain’t worth using.

When I turned to the problem of dealing with CBP, my main thoughts went to two devices that I use when traveling: my handy (mobile “phone”) and my laptop. I figured the easiest thing to do would be to just wipe them before coming into the country, but that presents some logistics problems.

For example, I could make a backup of my handy, copy it to a server at home, and then wipe it. The problem is that I have 64GB of storage on the device and I doubt I could transfer a backup in time over, say, a hotel Wi-Fi connection. One of my coworkers uses an iPhone and they thought about wiping their phone and just restoring it from iCloud when they were in the country, but then CBP could require that he turn over his iCloud password.

On my laptop I use whole disk encryption, but I thought about just rsync’ing my home directory and then deleting it before leaving, then again there is the WiFi issue and I really don’t want to have to deal with copying everything back when I’m home.

Then it dawned on me that if I didn’t know the encryption password, then I couldn’t reveal it. The problem became how to create a secure password that I couldn’t remember yet get it back when I needed it.

While my main desktop computer runs Linux Mint, I keep an old iMac on my desk mainly to run WebEx sessions and for those rare times I am forced to use a piece of software not available for Linux. It’s connected to the network, so I can access it remotely. But, if I can access it, I would be lying if CBP asked me for my password and I said I couldn’t retrieve it. Unlike the US Attorney General, I refuse to perjure myself.

Then it dawned on me that I could shut the iMac down remotely and have no way to turn it back on. Thus I could store a passphrase on it, retrieve it when I was back in the country, but until then I would be unable to unlock my devices.

That became the plan. So, the next time I’m returning from overseas, I’ll generate a new, random password. I’ll set that as the whole disk encryption password on my laptop and the encryption password on my handy (note that this is different from the screen-lock password). This will also tie up all of my social network passwords since I use complex ones and store them on those devices. Well, with the exception of my Google account, but since I use two-factor authentication I should be safe as my handy is the device that generates the codes (and I won’t carry any of the backup codes). As long as both of those devices stay powered on, I’ll be able to use them, but once I power them off they will be useless until I get to the office, power on the iMac, and retrieve the passphrase. Note that in order to do that, I’ll be firmly in the US and anyone who wants me to unlock my devices will need a court order.

Which I would respect, unlike CBP. I think the scariest part of the whole “Muslim Ban” incident was when CBP refused to honor court orders. America is built on three branches of government, and when the Executive branch ignores the orders of the Judicial branch we are all in trouble.

I had a two other problems to address, one of which is done. If I’m in the US but my handy is locked, how would I make calls? I might need to call my ride home, etc. To that end I bought a cheap “feature” phone and I’ll just move the SIM card to it when we land.

ZTE Feature Phone

The second issue is that while I should be on solid legal ground concerning my electronic devices, there is nothing preventing CBP from holding me for a long time. Thus the final step is to find an attorney and execute a G-28 form allowing them to represent me. I’m not sure if I need a civil rights lawyer or an immigration lawyer but I’m looking into it. My goal is to be able to notify my attorney when I am coming back into the country, and then send an SMS to them when I am through immigration. If that doesn’t arrive within two hours of my scheduled arrival, they need to come and get me.

I think the thing that bothers me the most about this whole process is the need for it. I’m not a tinfoil-hat conspiracy guy but the actions of the new government have me worried. As I use open source software almost exclusively I know I’m safer than most when it comes to surveillance, and I also don’t expect to run into any problems being an older, white male. But I’d rather be safe than sorry, and the only thing necessary for the triumph of evil is that good men do nothing.