Month: November 2017

Update on Expensify

Posted on by Tarus

I recently posted a rant on how a vendor we use, Expensify, appeared to be exposing confidential data to workers with the Amazon Mechanical Turk service. In response to the general outcry, they posted a detailed explanation on their blog.

It did little to change my mind.

So apparently what happened is that they used to use the Mechanical Turk from 2009 to 2012, so if you we a customer back then your information was disclosed to those third party workers. Then they stopped, supposedly using some other, similar, in-house system.

But, some genius there decided that the best way for certain customers to insure their receipts were truly private was to have them use the Mechanical Turk with their own staff. I covered that in my first post and it is so complex it hardly registers as a solution.

Of course, they decided to test this new “solution” starting the day before the American Thanksgiving holiday. This was done using receipts from “non-paying customers”. While we pay to use the service (not for much longer), if you were trying it out for free your receipts were exposed to Mechanical Turk workers. Heh, if you aren’t paying for the product you are the product. The post goes on to talk about the security of the Mechanical Turk service, which was surprising because they went on and on about how they didn’t use it.

What really angered me was this paragraph:

The company was away with our families and trying hard to be responsive, while also making the most of a rare opportunity to be with our loved ones. Accordingly, this vacuum of information provided by the company was filled with a variety of well-intentioned but inaccurate theories that generated a bunch of compounding, exaggerated fears. As a family-friendly business we try hard to separate work life from home life, and in this case that separation came at a substantial cost.

Well, boo hoo. If you truly cared about your employees you wouldn’t start a major beta test the day before a big holiday. I spent my holiday worrying about my employees’ personal data possibly being exposed through the Expensify service. Thanks for that.

What pisses me off the most is this condescending Silicon Valley speak that their lack of transparency is somehow our fault. That our fears are just “exaggerated”. When Ryan Schaffer posted on Quora that nothing personal is included on receipts, he demonstrated a tremendous lack of understanding about something on which he should be an expert. As they turn this new leaf and try to be more transparent, I noticed he deleted his answer from the Quora question.

Smells like a cover up to me.

Look, I know that being from North Carolina I can’t possibly understand all the nuances of the brain-heavy Valley, but if Expensify truly does have a “patented, award-winning” methodology for scanning receipts, why don’t they just make that available to their customers instead of using the Turk? This long-winded defense of the Turk seems like they are protesting too much. Something doesn’t make sense here.

I’ve told my folks to stop using SmartScan and that we would move away from Expensify at the end of the year. If you use, or are planning to use, Expensify you should deeply consider whether or not this is a company you want to associate with and if they will act in your best interests.

I decided the answer was “no”.

Dougie Stevenson – The Elvis of Network Management

Posted on by Tarus

David messaged me yesterday that Dougie Stevenson had died.

I hadn’t seen Dougie in person in a long time, but I’d kept up with him through the very networks he, in part, helped manage. While I had heard he wasn’t in the best of health, the news of his passing hit me harder than I expected.

I can’t remember the first time I met Dougie. I do remember it was always Dougie, rarely Doug and never Douglas. While most adults might drop such a nickname, it is a reflection on his almost childlike friendliness and good nature that he kept it. I do know that I was working at a company called Strategic Technologies at the time, so this would be the mid-1990s. I was working with tools like HP OpenView, and I’d often run into Dougie at OpenView Forum events. When he decided to take a job at Predictive Systems I followed him, even though it meant commuting to DC four to five days a week.

It was at Predictive that I got to see his genius at work. With his unassuming nature and down-to-earth mannerisms it was easy to miss the mind behind them, but when it came to seriously thinking about the problems of managing networks there were few who could match his penchant for great ideas. I used to refer to him as the “Elvis” of network management.

We were both commuters then. While he had lived in many places, he called Texas home as much as I do North Carolina. We were working on a large project for Qwest near the Ballston metro stop, and after work we’d often visit the nearby Pizzeria Uno. The wait staff loved to see Dougie, and would always laugh when he referred to the cheese quesadillas appetizer as “queasy-dillies”. This was back during the first Internet bubble, around 1999, and while many of us were working hard to make our fortune, Dougie never really cared that much for money. He used to joke it would all go to his ex-wives anyway. I know he had been married but we didn’t talk too much about that aspect of his life. He’d much rather talk about the hotrod pickup truck he was always working on when he had the time. I do remember he once walked away from a small fortune over principles – that was just the kind of person he was.

I can’t remember the last time I saw Dougie, but it could have been in Austin back in 2008. I have this really bad picture I took then:

Dougie and Me

Notice he has on his OpenNMS shirt. He never failed to promote our efforts to create a truly free and open source network management platform whenever he could.

As I’ve gotten older, I wish more for time than money. Between the business and the farm I’m kept so busy that I rarely get to spend as much time with the amazing people I know, and it would have been nice to see Dougie at least once more. In any case, a small part of him lives on in the hearts and minds of those who did know him.

Though it saddens me to say it, Elvis has left the building.

Expensify and Why I Hate the Cloud

Posted on by Tarus

Over the weekend I found out that Expensify, a service I use for my company, outsources a feature to Amazon’s Mechanical Turk service. Expensify handles the management of business expenses, which for a company like ours can be problematic as we do a lot of travel when deploying services. The issue is that the feature, the “smart scanning” of receipts, could potentially expose confidential data to third parties. As a user of Expensify, this bothers me.

Expensify touts “SmartScan” as:

As background, SmartScan is the patented, award-winning technology that underpins our “fire and forget” design for expense management. When you get a receipt, rather than stuffing it into your pocket to dread for later, just:

1. Take your phone out of your pocket
2. SmartScan the receipt
3. Put your phone back in your pocket

What they never told us is that if their “patented, award-winning technology” can’t read your receipt, they send it to the Mechanical Turk, which in turn presents it to a human being who will interpret the receipt manually. The thing is, we have no control over who will see that information, which could be confidential. For example, when I post a receipt for an airline ticket, it may include my record locater, ticket number and itinerary, all of which are sensitive.

This apparently never occurred to the folks at Expensify. Take this Quora answer from Ryan Schaffer, listed as Expensify Director of Marketing & Strategy:

Also, its worth mentioning, they don’t see anything that can personally identify you. They see a date, merchant, and amount. Receipts, by their very nature, are intended be thrown away and are explicitly non-sensitive. Anyone looking at a receipt isunable to tell if that receipt is from me, you, your neighbor, or someone on the other side of the world.

Wrong, wrong, wrong. It seems that Mr. Schaffer may limit his business expenses to the occasional coffee at Starbucks, but for the rest of us it is rarely that limited. For someone whose job is to perfect dealing with receipts, his view is pretty myopic.

For examples of what Expensify exposes, take a look at this tweet by Gary Pendergast.

Information Exposed by Expensify Tweet

It is also worth noting that it appears Expensify does its business on the Mechanical Turk as “Fluffy Cloud” instead of Expensify, which strikes me as a little disingenuous.

In a blog post this morning the company addressed this:

As you might imagine, doing this is easier said than done. Given the enormous scale and 24/7 nature of this task, we have agents positioned around the world to hand off this volume from timezone to timezone. Most of the US team is located in Ironwood, MI or Portland, OR (where we have offices and can train in person). Most of the international team is in Nepal or Honduras (where we work with a third-party provider to manage the on-site logistics). But regardless of the location, every single agent is bound by a confidentiality agreement, and subject to severe repercussions if that agreement is broken.

But if this were true, why are random people on Twitter announcing that they can see this data? Are they relying on the Amazon agreement with the people working as part of the Mechanical Turk? That doesn’t instill much confidence in me. But then in the same blog post they double down, and suggest that if you want extra security, you can just set up your own staff as part of the Mechanical Turk:

1. You hire a 24/7 team of human transcription agents.
         o For the fastest processing we suggest staffing three separate shifts — or daytime shifts in three different offices around the world. Otherwise your receipts might lag for many hours before getting processed.

2. They apply to Amazon Mechanical Turk for an account. Be aware that this is a surprisingly involved process, including:
         o The agent must sign up using their actual personal Amazon account. If your account doesn’t have an adequate history of purchases (each of which implies a successful credit card billing transaction and package delivery) or other activity, you will be rejected.
         o The agent must provide their full name, address, and bank account information for reimbursement. Amazon verifies this with a variety of techniques (eg, confirm that your IP is in the country you say you are, verify the bank account is owned by the name and address provided, full criminal background check), and if anything doesn’t add up, you will be rejected.
         o Rejection is final. It requires such an abundance of verifiable documentation (most notably being an active Amazon account with a long history) that you can’t just create a new account and try again.
         o There is no apparent appeals process. Accordingly, I would recommend confirming before hiring that the candidate can pass Amazon Mechanical Turk’s many strict controls because we have no ability to override their judgement.

3. You notify us of the “workerID” of each of your authorized agents.
         o Though you are not obligated to share your staff’s identity with us directly, your staff will still be obligated to follow the Expensify terms of services. Failure to comply with our terms will result in an appropriate response, starting with immediate banning by our automated systems, ranging up to our legal team subpoenaing you (or failing that, Amazon) for the identity of the agent to press charges directly.

4. We will create a “Qualification” for your “Human Intelligence Tasks” (HITs) that ensures only your agents will see your receipts.

5. Your staff will use the Amazon Mechanical Turk interface to discover and process your employee’s receipts.

That’s the solution? This is what passes for security at Expensify? Hire three shifts of employees all using verified personal Amazon accounts and then you can be sure your confidential data is kept confidential?

Wouldn’t it just be easier to create a small webapp that would present receipts to people in a company directly without going through the Mechanical Turk? Heck, why not just bounce it back out to you – it isn’t that great of a chore.

Plus, basically, if you don’t do this Expensify is saying they can’t keep your information secure.

This is what frustrates me the most about “the Cloud”. Everyone is in such a rush to deploy solutions that they just don’t think about security. Hey, it’s only receipts, right? Look what I was able to find out with just a discarded boarding pass – receipts can have much more information. And this is from a company that is supposed to be focused on dealing with expenses.

I demand two things from companies I trust with our information in the cloud: security and transparency. It looks like Expensify has neither.

I will be moving us away from Expensify. If you know of any decent solutions, let me know. Xpenditure looks pretty good, and since they are based in the EU perhaps they understand privacy a little better than they do in San Francisco.

2017 Cubaconf

Posted on by Tarus

I’ve just returned from Cubaconf in Havana, which was also my first visit to Cuba. It was a great trip and I’ve got enough material for at least four blog posts. Most of them won’t deal with free and open source software, so I’ll put them up on my personal blog and I’ll add links here when they are done..

Cubaconf is in its second year, and while I really wish they would have called it “Cuba Libré” (grin) it was a good conference.

There is a spectrum within the Free, Libré, Open Source Software (FLOSS) community, and this is often described by trying to separate the term “open source” from “free software”. If we define “open source” as any software with a license that meets requirements of the Open Source Definition (OSD) and “free software” as any software with a license that meets the requirements of the Four Freedoms, they are the same. You can map the ten requirements of the OSD onto the four requirements of free software.

Open Source is Free Software Chart

However, it can be useful to separate those who look at FLOSS as simply a development methodology from those who view it as a social movement. When companies like Microsoft and Facebook publish open source software, they are simply looking to gain value for their business that such sharing can create. It’s a development methodology. When people talk about free software, they tend to focus on the “help your neighbor” aspect of it, and this was more the focus of Cubaconf than simply creating new code.

The main thing I discovered on my visit was that Cubans face severe limitations on many things, but I’ve never met a people more determine to do as much as they can to make their situation better, and to do it with such passion. If I had to pick a theme for the conference, that would be it: passionate continuous improvement.

Cubaconf Registration

The three day conference had the following structure: Day One was a standard conference with keynotes and five tracks of presentations, Day Two had keynotes and more of a “barcamp” organization, and Day Three was set aside for workshops, as well as the obligatory video game tournament.

They did have the best lanyard sponsor I’ve seen at a technology conference:

Cubaconf Lanyard and Badge

I was in Cuba with my friend and coworker Alejandro, who used to live in Venezuela and is a fluent Spanish speaker, and Elizabeth K. Joseph, who promotes the open source aspects of Mesosphere. We shared a three bedroom “casa particular” in Old Havana, about a ten minute walk from the conference, which was held on the second floor (third floor if you are American) of the Colegio de San Gerónimo.

Everyone was together on the main room for the first keynote.

Cubaconf People in Room

While both English and Spanish were spoken at the conference, the presentations were overwhelmingly in Spanish, which was to be expected. I can get by in Spanish, but the first speaker, Ismael Olea, spoke fast even for the native speakers. At least I could understand most of the content in his slides.

Cubaconf Ismael Olea

Olea is from Spain and did a keynote on HackLab Almería. Almería is a province in the southeastern part of Spain, and with a population of around 700,000 people it is much smaller than provinces like Madrid (6.5 million) and Barcelona (5.5 million). As such, the region doesn’t get as much attention as the larger provinces, and so they goal of Hacklab Almería is to use technology at the “hyperlocal” level. They define themselves as a “collective of technological , social and creative experimentation” and FLOSS plays a large role in their mission.

After the keynote, we broke up into individual sessions. I went to one called “How to Make Money with Free Software” presented by Valessio Brito from Brazil. While he spoke mainly in Spanish, his slides were in Portuguese, but I was able to follow along. His presentation focused on how he used his knowledge of FLOSS to get consulting engagements around the world. This was pretty topical since in Cuba, as elsewhere, having strong software skills can be lucrative, and since a lot of proprietary software is either impossible to get or too expensive, having skills in open source software is a plus.

Cubaconf Valessio Brito

Also, I liked his shirt.

Our OpenNMS presentation was in the next time slot. I asked our hosts if they would like the presentation in English or Spanish, and when they said Spanish I asked Alejandro to give it. He did a great job, even though he had only a short time to understand the slides.

Cubaconf Alejandro Galue

The lunch break came next, and we walked a couple of blocks to the Casa de Africa, a museum dedicated to the African influence in Cuban culture.

Cubaconf Lunch Break

We ate sandwiches and talked out on the patio. This would be the location for lunch for all three days.

Cubaconf Maira Sutton

After lunch I watched a presentation by Maira Sutton called “Fighting Cyber Dystopia with Tech Solidarity and the Digital Commons” which is a long way to express the idea of using free software combined with working together to take back some of the power from large corporations. Her main example talked about the city of Austin, Texas, and its fight with Uber and Lyft. Austin wanted all ride sharing drivers to have to undergo a background check that included fingerprints. Sounds reasonable, but Uber and Lyft resisted, eventually leaving the city.

However, services like Uber and Lyft can be downright useful, so a number of startups filled in the gap, offering similar services that met the City’s fingerprint requirements. Instead of acquiescing to local laws, Uber and Lyft took their fight to the State legislature, which overturned Austin’s regulation.

Even though it is a sad ending, the example did demonstrate that combining technology and social action can result in solutions that can meet or exceed those provided by large commercial companies.

Cubaconf First Night's Event

For each night of the conference there was an event, and the one for Tuesday was held at a modern art gallery on the southern side of Old Havana. There was lots of food and drink, and I got exposed to a project called cuban.engineer. This is a group to promote technology careers within Cuba, and I had seen their shirts at the conference.

Cubaconf cuban.engineer shirts

In a lot of the world we take Internet access for granted. I can remember accessing the Internet from the night market in Siem Reap, Cambodia, on an open wi-fi connection. That doesn’t exist in Cuba. Cuba is one of the most disconnected countries in the world, which can make working with technology difficult. Access is controlled by an agency called ETESCA. To access the Internet you purchase a card which offers a certain number of hours of use, and then you have to locate an area with a wi-fi hotspot (usually near a park). The card has a number of digits for a username and a number of digits as a password, and once you get connected you hope you stay connected long enough to do what you need to do.

No one is exempt from this. Even in our apartment the owner would use one of these cards to enable access for the hotspot on the ground floor. So, if you are a technology business in Havana, your first job is to located your office near a hotspot, and then buy a bunch of these cards.

Thus you can imagine that sharing in a big part of the culture. People burn and swap CDs with software such as Ubuntu on them, and they tend to use Gitlab to make local mirrors of code repositories. While wi-fi equipment can be hard to come by, people have been able to set up their own, private wi-fi networks within cities like Havana to make sharing easier. There is no Internet access (I joked that it was Cuba’s “dark web“) but they can set up tools like Rocket.Chat to communicate and share.

Despite limitations in acquiring software, Microsoft Windows is still the most common operating system running on Cuban computers. An attempt was made to create a Cuba focused Linux-based distro called Nova. I was told that they even experimented with making it look as close to Windows 7 as possible, but people were still tied to using Windows. According to Wikipedia this distro is no more, which is a shame.

Cubaconf Mixæl Laufer

The second day started with the meter pegged at full on social justice, with a presentation by Dr. Mixæl S. Laufer, Ph.D., from Four Thieves Vinegar. They are a collective aiming to share information on how to create pharmaceuticals in places where they might not be available. If you live in the US than you probably heard of Martin Shkreli who as CEO of Turing Pharmaceuticals raised prices 5600%, and EpiPen maker Mylan who raised the price of this life saving device several hundred percent just because they could. Laufer showed how you could make your own EpiPen for around $30, among other things.

Now drug companies will say that they have to charge that amount to cover the costs of creating new drugs, but the EpiPen greed demonstrated that wasn’t true. Running health care as a “for profit” enterprise has always seemed inherently wrong due to the incentives being more toward making money versus keeping people healthy, but that is commentary for another time and place.

I had to leave after that presentation for something special. I make classic cocktails as a hobby, and one of our hosts asked me to speak to a school for bartenders (cantineros) on the great tradition of Cuba cocktails. It was a blast and I’ll write that up soon.

Cubaconf Wednesday Event

Wednesday night’s event was held, appropriately, at a bar in an area called Barber’s Alley. It was a fun gathering and I got a nice picture of some of our hosts.

Cubaconf Hosts

Left to right is Jessy, Pablo, pb, and Adalberto.

There was also a guy there who made pipes, specifically replicas of native American peace pipes, and one was passed around.

Cubaconf Peace Pipe

The third and final day was a series of workshops, but was started with a keynote from Ailin Febles, from the Uniōn de Informāticas de Cuba, a non-profit organization to bring together “all technicians, professionals and people related to information and communication technologies in a space that enables mutual support of the associates in the achievement of their professional, academic, scientific, cultural and personal objectives”.

Cubaconf Ailin Febles

Of course, a lot of their organization is driven by open source software.

Cubaconf Software used by UIC

I hope they switch to Nextcloud from Owncloud soon.

There was one morning workshop in English, ironically by a German named Christian Weilbach, on machine learning. I was interested in the topic since I keep hearing about it lately, and the fact that I would probably be able to understand it was a plus. To me machine learning is magic, and I wanted to dispel some of that magic.

Cubaconf Christian Weilbach

It worked. It turns out that machine learning is, to a large extent, what we used to call linear algebra. It just is able to work on much larger and more complex data sets. I’m still eager to play more with this technology, but it was nice to learn that it really isn’t all that new.

Cubaconf Old Car Taxi

After lunch we decided to spend our last afternoon exploring Havana a bit.

Cubaconf Brewery Event

The final evening event was in a brewery, and I enjoyed the beer. What I enjoyed more was the opportunity to talk with Inaury about race in Cuba. Cubans come in all shapes and sizes, from people with light skin, blond hair and blue eyes to people so dark they are almost blue, yet they all seem to interact and socialize with each other more so than any other place I’ve been. I plan to chat more about that in a blog post as well.

Overall I had a great time in Cuba. I love the fact that working in free software means I can make new friends in almost any country, and that even a place with limited resources can put on a great conference. If you get a chance to go to Cubaconf, you should take it.