While I love living “out in the country” I often envy my urban friends for their network connectivity. When I moved out to the farm in 1999 the only “high speed” access was satellite, and even that required a modem and a phone line. I was overjoyed when Embarq finally deployed DSL to my house, and while 5bps down might not seem like much these days it seemed heaven-sent back then.

Jump forward 20 years and Embarq became Centurylink which is now Brightspeed. I had a pretty poor opinion of Centurylink (or as I called them CenturyStink) but high hopes for Brightspeed when they bought Centurylink’s ILEC business in our area, but they have been disappointing. Here is that story.

Both my wife and I work from home, and when our DSL circuit is working it works well enough for us to get our jobs done. At 11Mps down and 640kbps up it doesn’t even qualify as “broadband” but it is a trade off I’m willing to make in order to live where I live.

Starting back in early November we began to have issues with the quality of the DSL connection. Quality issues are always frustrating since the support technicians at the provider never seem to have the tools to properly measure it. Instead they just tell me the circuit is “up” so I should be satisfied, even though I tell them that while it is up, it is unusable.

The issue was high latency and packet loss. Latency is a measure of the time it takes information to travel through the network and packet loss indicates that some of that information never makes it to its destination. The protocols used in networking will automatically deal with packet loss by sending the information again, but the more this happens the worse the experience is for the user. Things that can handle packet loss gracefully like e-mail, web pages and chat just seem very slow, while anything that requires a more steady flow of information like video or gaming just don’t work at all.

Having done network monitoring for much of my professional life, I monitor the quality of my DSL circuit by attempting to reach the 8.8.8.8 IP address, which is a highly available DNS server run by Google. Here is a recent graph:

Now normally the graph should be green and pretty much focused around 45ms. This one was all over the place. I asked my neighbor to execute a ping to the same IP address and her connection was working fine, so I assumed it was an issue specific to me.

Trying to get support from Brightspeed was very frustrating. As I mentioned above they just tend to tell me everything is okay. I even reached out to one of my LinkedIn contacts who is an executive vice president at Brightspeed for help, and I think he was responsible for a ticket being opened on November 7th in the “BRSPD Customer Advocacy (Execs)” queue. I really appreciated the effort but it didn’t help with getting my issue resolved.

Just after Thanksgiving I called again and I was told that the problem was that my modem was wrong and doesn’t work with the DSL circuit we have, even though it was the model they sent to me and it had been working fine up until November. In any case they said they would send me a replacement and it would arrive in two days.

Ten days later I call back and ask, hey, where is my modem? They told me it was still “in process” but since I’ve been waiting so long they’ll overnight one to me. It shows up the next day and of course doesn’t fix the issue, but it has newer software than my other modem and reports the status of the DSL circuit. On the web page of the device this is usually represented by the word POOR in dark red, but sometimes it would improve to MARGINAL in a slightly lighter red. I call and explain this to Brightspeed, and after dealing with this for over two months they agree to send out a technician in two days.

When I finally get the e-mail confirming the appointment, it is for the following Thursday, January 5th, eight days away. That also happens to be when we are closing on a new house, so I can’t be here to meet with them. For some appointments they show up early, so I didn’t change it right away, but when they hadn’t shown up by Tuesday I decided to reschedule it.

I went to the e-mail and clicked on the link to reschedule and got sent to the Centurylink site. Of course they wanted me to confirm my account, but nothing I typed in: phone number, e-mail, or account number, worked because I no longer have a Centurylink account. (sigh)

[Note: it looks like this has been corrected, finally, on the Brightspeed website. Not sure about links in e-mails]

In the process I did find out that my Brightspeed account number ends in “666” so perhaps that is indicative of something.

I eventually ended up calling support once again. I believe it would take the average caller about six minutes to reach a human through their system, as it prompts you for a variety of things before allowing you to speak to a person, but I had been calling for over two months so I can speedrun the thing in about four and half minutes by pressing buttons before the prompt is finished.

The person I talked to about rescheduling the appointment kept me on hold for about 30 minutes before telling me that the whole dispatch system for technicians was down and that she would call me back in two hours. She never did.

The next day I made one more attempt to reschedule the appointment, but was told that the next available appointment was so far out in the future that I should just keep it, since the technician won’t need to enter the house. I left a long letter taped next to the demarcation box on my house with a detailed description of the problem, and hoped for the best.

Unfortunately, they sent out Brandon. To my knowledge there are only two technicians assigned to our rather large county: Brandon and Elton. I much prefer working with Elton since Brandon doesn’t really seem to be the kind of person who does a deep dive into the problem, but I recently learned that Elton has moved into the back office and wasn’t doing service calls anymore.

As I feared, Brandon marked the issue as closed without fixing it. Once again into the support phone queue, where I was told that he had run a test “for five minutes” and my circuit was fine. (sigh)

I did get a text asking if my problem was resolved, to which I said “NO!” and I was later contacted by a person from Brightspeed to follow up. After a very long conversation she offered to send someone else out, and that person arrived yesterday.

Philip, who is based out of Wake County (one county to the east of us) showed up promptly at 8am and within ten minutes had diagnosed a grounding issue with the wires coming to our house. In about 45 minutes he had repaired it, but he warned me that there was also an outage in the area which would explain my now 900ms ping times (but no packet loss). I trusted him that it would eventually resolve and about 30 minutes later things were much, much better.

You can see where the network was bad before Philip showed up, the gap where he was working on the system, and then the return to a more expected quality of service.

It still isn’t perfect. I’m seeing a lot of jitter from time to time which is indicated by the spikes, but for the most part the user experience is fine. I was able to participate in our departmental weekly video call without issue yesterday for the first time in months.

And that’s was really bothers me the most. For nearly three months Brightspeed was gaslighting me that my service was fine, when, as most IT professionals would expect, it turned out to be a physical layer problem. In retrospect it makes sense since we’ve been having an especially wet winter and that would have caused the grounding issue to be amplified.

I figure I spent between 40 and 60 hours actively involved in getting this addressed, and that is time I’ll never get back.

Of course it could be worse. The local newspaper published a story about a community in Chatham County that was without service from Brightspeed for a total of 51 days. At least our connection was usable enough that it only required a few trips to the public library for access during important deadlines.

There is some good news in that same newspaper issue that some attempts are being made to help those of us in rural areas get broadband. Some of you may be thinking Starlink, but I was on their waiting list for two and a half years without getting my equipment and when they pushed it out to late 2023 I just gave up and asked for my deposit back.

I am not a huge “we need to regulate everything” kind of guy, but broadband has become one of those services that is so important and that the free market has failed to provide that I would welcome government involvement in getting this issue addressed. But so far the communications lobby has been strong enough to prevent any kind of oversight, so I won’t hold my breath.

I recently posted a rant on how a vendor we use, Expensify, appeared to be exposing confidential data to workers with the Amazon Mechanical Turk service. In response to the general outcry, they posted a detailed explanation on their blog.

It did little to change my mind.

So apparently what happened is that they used to use the Mechanical Turk from 2009 to 2012, so if you we a customer back then your information was disclosed to those third party workers. Then they stopped, supposedly using some other, similar, in-house system.

But, some genius there decided that the best way for certain customers to insure their receipts were truly private was to have them use the Mechanical Turk with their own staff. I covered that in my first post and it is so complex it hardly registers as a solution.

Of course, they decided to test this new “solution” starting the day before the American Thanksgiving holiday. This was done using receipts from “non-paying customers”. While we pay to use the service (not for much longer), if you were trying it out for free your receipts were exposed to Mechanical Turk workers. Heh, if you aren’t paying for the product you are the product. The post goes on to talk about the security of the Mechanical Turk service, which was surprising because they went on and on about how they didn’t use it.

What really angered me was this paragraph:

The company was away with our families and trying hard to be responsive, while also making the most of a rare opportunity to be with our loved ones. Accordingly, this vacuum of information provided by the company was filled with a variety of well-intentioned but inaccurate theories that generated a bunch of compounding, exaggerated fears. As a family-friendly business we try hard to separate work life from home life, and in this case that separation came at a substantial cost.

Well, boo hoo. If you truly cared about your employees you wouldn’t start a major beta test the day before a big holiday. I spent my holiday worrying about my employees’ personal data possibly being exposed through the Expensify service. Thanks for that.

What pisses me off the most is this condescending Silicon Valley speak that their lack of transparency is somehow our fault. That our fears are just “exaggerated”. When Ryan Schaffer posted on Quora that nothing personal is included on receipts, he demonstrated a tremendous lack of understanding about something on which he should be an expert. As they turn this new leaf and try to be more transparent, I noticed he deleted his answer from the Quora question.

Smells like a cover up to me.

Look, I know that being from North Carolina I can’t possibly understand all the nuances of the brain-heavy Valley, but if Expensify truly does have a “patented, award-winning” methodology for scanning receipts, why don’t they just make that available to their customers instead of using the Turk? This long-winded defense of the Turk seems like they are protesting too much. Something doesn’t make sense here.

I’ve told my folks to stop using SmartScan and that we would move away from Expensify at the end of the year. If you use, or are planning to use, Expensify you should deeply consider whether or not this is a company you want to associate with and if they will act in your best interests.

I decided the answer was “no”.

Here is a rant about Time Warner/Charter/Spectrum or whatever the heck they call themselves these days. It illustrates how this large company can have a huge negative impact on a small business, and why treating Internet providers as common carriers is so important.

Our company wouldn’t exist without the Internet. Outside of the fact that our products are mainly used to monitor Internet resources, we host a number of servers from our office and about half of the staff works remotely so we rely on the Internet to communicate and coordinate.

Back in 2012 I contracted with Time Warner to provide Internet access to our office. We had fiber to the building and while our service was considerably more expensive than coax, I liked the fact that it was symmetrical and expandable. We started of with 20 Mbps but soon increased that to 50 Mbps. Over five years we only had one outage, due to a misconfiguration of our Customer Premise Equipment (CPE), and they corrected it within 20 minutes. I love the fact that when you called in the person who answered the phone understood terms like “duplex” and they were always very helpful.

Note the scenario: happy customer who is happy paying a premium for enterprise-level service.

Now let me tell you why all that goodwill has gone away.

Earlier this year we decided to move our office from Pittsboro, NC to Apex, NC. The first thing I did was contact Time Warner (well, Charter at the time) to insure that they could provide fiber to the new location. They said they could, although it would take 45 to 60 days. As our new office space needed to be completed, we were targeting an April 1st move in date anyway, so on February 15th I placed the order for the new service. At best, it would be available on the 1st and at worst it would be ready by the 15th. We told the old landlord we’d be out by April 30th just in case and to give us more time to move.

Finally, Spectrum doubled our speed and cut the price in half. I was feeling pretty good about the whole thing.

The feeling didn’t last.

As we got closer to April, things started to go wrong, most of it due to the fact that Spectrum is now such a behemoth that they have no idea what they are doing. In order to get fiber into our new building, they needed what is called a “Right of Entry”. They sent it to our landlord who promptly completed the form and sent it back. However, that person didn’t let the project manager know the form had been received, so he did absolutely nothing. Ten days (!) later I get a note that our build out had been suspended because of the lack of the ROE form. A form, I should point out, that was sent to them, twice.

At the end of March I’m told that our new date is May 11th. I’m unhappy – due to their poor processes I now have a new office that I can’t use for six weeks (remember, we took possession and started paying rent on April 1st). We also had to be out of the old office by the end of April. Luckily I work with a great team that is able to be productive when working from home, so I decided to suck it up and live with it.

On April 12th I get an update – the new date for the end of construction is now May 15th due to processes within Spectrum taking too long to finalize the work with a contractor. Now the actual date we’ll have Internet has been pushed out to the week of May 29th.

I am livid. By this point I’m ready to switch to the other option, AT&T. Unfortunately, they also need 45 to 60 days for service installation so I realize at this point I’m stuck with Spectrum.

I ask my salesperson for options and he suggests we get coax installed for a month (for a fee, of course). Since our office is right next to a large housing development they can get coax in the following week. I sign off on it.

It didn’t happen. When May arrived some of us started working in the new office mooching off the neighbor’s Wi-Fi from AT&T (with permission of course). I ended up traveling for a couple of weeks so I completely forgot about the coax option (it’s not like Spectrum was keeping me updated on anything – I’d have to reach out to them for an update). I did get a note on May 10th that all construction had been completed for the fiber and another note on May 18th that our new install date was June 2nd.

(sigh)

So, 45 days late, we have a firm install date. Wonderful.

Imagine how I felt when on the 24th of May I received a note that more construction was needed and that it would be pushed out another 30 days at least. When I get extremely angry I refer to it as going “non-linear” as that how fast my blood pressure rises. As I was ranting to pretty much everyone I’d ever interacted with at Spectrum it dawned on me that this could be for the coax order. Turns out that was the case. Apparently our crack project manager on the coax side decided to route our service from a point several miles away instead of from the one nearly across the street. This is why it was delayed and why the construction was needed. By this time we are about a week out from having fiber so I canceled the order. I did get a very apologetic call from the coax salesperson which I appreciated (under Spectrum, fiber [Enterprise] is handled by one sales team and coax [Business] is handled by another), and I made it clear that I’d be okay with everything as long as the fiber was delivered as promised on the 2nd.

It was. Around noon on June 2nd we had our 100 Mbps service and on the 3rd we moved all of our devices from the old office in Pittsboro to the new one in Apex. I informed my salesperson that they could disconnect the old service and despite all of the problems, I was happy with the new service.

So the whole process cost me two months rent and a few years off my life, but it was finally over.

Not so fast – the other shoe fell today.

I get an e-mail that I need to confirm my disconnect request. That didn’t bother me, in fact I appreciated it, but what did bother me was an additional note that it would be done within 30 days. When I replied I asked for clarification – would I be *paying* for the service I wasn’t using until they could disconnect it? The answer was “yes”.

I experienced a new word – apoplectic.

Due to the fact that the bureaucracy behind the new merged Spectrum company is so bad, I’m out nearly ten thousand dollars. That is the real money – it’s probably cost us twice that again in lost productivity from lack of network access and dealing with them throughout this process. We’re not one of those companies that is too big to fail so this really impacts us negatively. Had it been explained to me that I’d have to pay for the service until it was disconnected, I would have put the disconnect order in a month ago, but then had I used the date I was originally promised, our servers would have been off-line for over a month. That would have been catastrophic to our company.

Finally, I’ve gone from a happy customer to an extremely pissed off one who will be actively looking for options. Based on my experience I would suggest any business looking for network access look elsewhere.

Access to the Internet has become as important as other utilities such as electricity, water and sewer and just like those utilities it needs to be regulated as one. This is why the decision by the new industry-picked head of the FCC to reverse the decision to classify Internet access under Title II as a “common carrier” is so devastating to businesses like mine. Our company is small, yet we put millions of dollars into the local economy each year. You multiply that by the number of other small businesses and it can have a great impact to any community. Barriers put up by companies like Spectrum demonstrate that they can’t self-regulate and the government needs to take a firmer hand (and this is coming from a left-leaning libertarian).

I will be protesting that final bill for Internet access and I would welcome any advice on how to deal with a company like Spectrum. Let’s hope that there is a change soon so that other businesses can focus on creating value and not have to deal with the crap we had to endure.

I’m not holding my breath.

Recently, I was talking to a person in the tech industry and describing all of the cool things we are doing with OpenNMS, when he kind of cut me off and went “Oh, monitoring? Meh.”

Well, I can’t remember if there was an actual “meh” but that’s how it came across, and I’m afraid the reaction is probably more common that I would think. Monitoring isn’t sexy, but it surprises me that people can’t see how critical it will be to the future of any business.

While forecasts vary, by 2020 there are expected to be over 30 billion devices on the Internet, and that figure will skyrocket to over 75 billion by 2025. Just knowing what is connected to your business network is going to become critical, as well as making sure it belongs there in the first place and, if so, is functioning properly.

Outside of the obvious security concerns, as people began to transact business more and more through devices rather than people, faults in those devices will directly impact revenue as people search for other options when faced with a bad experience.

Here are a couple of examples.

One of the greatest inventions in my lifetime is the ability to buy fuel at the pump. You just pull up, swipe your card, pump and then leave. You used to have to pay inside, and some places made you pay first which meant two trips in if you were paying by credit card. It could be cold or rainy, and not only did you have to wait in line behind people buying food or lottery tickets, you had to leave your car out by the pump possibly blocking the next customer.

The only problem I’ve experienced with this process concerns the receipt. Quite frequently I need a receipt, but it seems the pumps I choose are always out of paper. The little red indicator mark when the paper roll is almost finished isn’t visible to the cashier since there really isn’t one out by the pump. It is frustrating, but it is not like I have a choice at the moment. If there was some way to monitor the pump for a “low paper” alarm, it would improve my shopping experience.

One shopping experience that did result in my leaving the store without a purchase happened yesterday at a Lowe’s Home Improvement store. I needed some florescent lights for the new office so I went by on my way home. I picked up four bulbs (two that I needed and two spares) and went to the checkout area.

I walked past several unmanned cash registers until I got to the “Self Checkout” section, which was the only thing open. Of the four machines, two had red blinking lights on them (that are green when things are functioning normally) and the one lone, overworked cashier was doing her best to help people out. I usually don’t mind using Self Checkout and when I noticed one of the two machines was open (everyone else was waiting for the attention of the lone cashier) I went to it and started my purchase.

I scanned my “My Lowe’s” card and then the first bulb. “Eight ninety-five” piped up the voice and I placed it in a bag.

Here is where the problems started. First, I hate the fact that with these Self Checkout kiosks they don’t trust you to use a “quantity” key. I was buying four identical items but I was required to scan each one. Next, the bulb was light enough that it didn’t register as having been bagged, so the interface yelled at me and presented me with a button marked “Skip Bagging Item?”.

I sighed and, having no other option, hit the button. I then went on to scan the next three bulbs. However, as I bagged the fourth bulb, the scale must have started working since the whole unit went into some kind of alarm mode, screeching “Unidentified Object in the Bagging Area!” and the screen was locked until the cashier had time to come and fix it.

I looked around the area, and by this time all four kiosks had a flashing red light, there were at least three shoppers lined up to use them in addition to those of us already there, and our valiant cashier was busy helping a guy ring up his plumbing supply purchase which consisted of a ton of small copper fittings which most likely wouldn’t be registered by the scale.

I gave up. I picked up my bulbs and returned them to the Lighting section, passing three employees in the customer service area helping zero customers. Before I reached the car I’d ordered the same bulbs on Amazon at a fraction of the price, and they’ll be here on Friday.

Yes, I’m complaining, but how could monitoring have helped here? First, there is some sort of monitoring – those little red lights. When they all light up you would assume someone, or perhaps multiple someones, would come by to help. A monitoring system could have made sure that happened by using an additional notification system outside of the lights, and escalating it until the problem was addressed.

A more long term solution would be to collect information on the purchasing experience and the problems people encountered and to make changes to the automated kiosk software. I’m certain that Lowe’s didn’t write that software but instead bought it, and like most proprietary software solutions they now have to fit their processes to the application instead of the other way around. It probably wasn’t designed for a store that sells a lot of small, light things which is central to the issues I have using it.

With the rise of IoT devices, robotics and other forms of automation, monitoring is going to become extremely important. Lowe’s lost out on a $40 sale, but think of something like an assembly line where a problem could result in the loss of thousands of dollars a minute. Our goal at OpenNMS is to be ready for it, and to build products that make people go “Monitoring? Oh yeah!”.

The last 18 months of my life have been delightfully free of “open core” companies. These were companies who pretended to be “open source”, at least in their marketing materials, yet their business model was based on selling “enterprise extensions” which consisted of proprietary software that actually had the features you wanted. Basically, the open source piece was a loss leader to get you to buy the commercial edition, and as Brian Prentice pointed out so eloquently there was no real difference between “open core” and traditional closed source software. We like to call these businesses “fauxpen source“.

Customers realized this as well, which lead most open core companies to switch their tactics. While many still maintain an open source project, they have removed the term “open source” from their websites and most of their marketing (often replacing it with “open architecture”). I’m happy with this, as it allows true open source companies like OpenNMS and Nextcloud to differentiate ourselves while allowing these other companies to still produce open source software without misleading the market.

But lately I’ve been introduced to two new licenses that offer access to the source code without meeting the ten requirements of the Open Source Definition. These licenses further muddy the waters due to giving access to the code without including the freedoms of truly open software.

The first case was from Monty Widenius, who announced a proprietary Business Source License (BSL) for some of the MariaDB products. Monty was the guy who earned €16.6 million by selling MySQL to Sun and then got upset when Sun got bought by Oracle. Apparently, he seems to be unhappy that he isn’t earning enough money from his fork of MySQL products so he wants to create commercial software but not call it that.

The BSL, or as I call it, the “Rape of Large Companies License” allows the developer to offer the code up for use for free unless you cross some sort of arbitrary threshold, also set by the developer. In three years that code will revert to an OSI approved license, in this case the GPLv2, and if you are above the usage threshold then you don’t have to pay anymore.

I’m not sure what his goals are here, outside of running a commercial software business while paying lip service to open source software. Perhaps he hopes to get people to contribute to BSL licensed projects as long as their use case is small enough not to cross the “pay me” threshold, but more likely he just wants to ride on the coattails of the success of open source software without committing to it.

I learned of another such license called the Fair Source License (FSL) from a post by Ben Boyter who writes the Searchcode Server. Ben, at least, is a lot more up front about his reasons for adding a “GPL Timebomb” to his code. Initially, the code is published under the FSL but with a switch to the GPLv3 in three years. He isn’t expecting contributions and instead has offered up the code simply so it can be audited for back doors. As this is one of the more powerful features of open source software I applaud him for doing it, but I really wish he hadn’t used the term “bomb”. I have to deal with terms like “GPL poisoning” enough in my business that negative words like that just tend to scare people. He should have called it “Happy Fun Lucky GPL Gift Giving Time!”

Look, I’m all for anything that gets more code out there under an OSI-approved license but, c’mon, three years is a lifetime in this industry. Enterprise customers, who would be most affected by this license, will still have to approach the buying decision as if a BSL or FSL licensed application were commercial software. Even with the three year revenue window, it is unclear what happens if, say, a huge security bug is discovered three years out. Does the code to fix that bug restart the clock?

The whole process is confusing and doesn’t help the cause of open source software. I think open source is awesome and extremely powerful, and when I see things like this I’m almost insulted, as if the developer is saying “when I’m done you are welcome to my leftovers”. Instead of announcing a future switch to an open source license years in advance, they should just open it when they are ready, like id Software does with the Doom engine.

I’m giving a talk at All Things Open about running a truly open source business, the core point of which is that you can’t have an open source business with a business model based around selling software. No matter how you dress it up by either calling it “open core” or “business source” it is still proprietary software.