This is the first conference since joining AWS that I have booth duty, so I won’t be able to spend as much time in the sessions as I would like, but I did want to catch one of the first sessions of the day which was “Speedrunning Kubernetes”.

The main reason I wanted to see this talk was to see Kat Cosgrove in action. Prior to coming to AWS I didn’t know about her but I ended up following her on Twitter and found that she has strong opinions, and I tend to like people who have strong opinions. I figured the presentation would be entertaining and that I might learn something.

I wasn’t disappointed.

The title alludes to a “speedrun” which is an attempt to complete a video game as quickly as possible. The goal of this talk was to bring up a working Kubernetes cluster as if you were doing a speedrun. It also included one of the more … unusual … analogies I’ve seen in a technical presentation (including my own) by using a Chihuahua as a metaphor.

If the goal is to provide the “cheeseburger” application, consisting of the bun service, the patty service, the cheese service, the mustard service, etc., each instance of the application (i.e. each burger) can be considered a “pod”. There are two pods under each foot of the dog representing two-pod “nodes” and the dog forms the control plane.

Remember, now that you’ve seen it, you can’t unsee it.

That was the only session I made on Day Two, but I did get some time to wander around the Exhibit Hall. The Software Freedom Conservancy had a booth, and since they are one of my favorite organizations I stopped by to chat with Pono Takamori. I know a number of folks that work there and they serve as almost a reference implementation for trying to live using 100% free software. Pono was telling me that it was getting almost impossible to find a totally free mobile wireless solution since 3G went away, as all of the modern modems tend to use binary blobs.

Now, when these exhibit halls are being set up, the “booths” are laid out with little generic signs showing the owner of the booth, and most of the time they eventually get covered up once the booth is complete.

I know the Sun acquisition was a long time ago, but I still get cognitive dissonance when I see a MySQL sign next to an Oracle one.

The AWS booth for this conference is really awesome. I bow down to the genius that is Spot Callaway, and he pitched a booth design that was to invoke a teenage geek’s basement, where one might play video games and Dungeons and Dragons (think Stranger Things). The walls of the booth are made to look like brick, and there are chairs, a couch and an SNES console emulator.

The featured AWS project for this conference is Bottlerocket, and I got to learn a bit about it and meet members of the team. Bottlerocket is a minimal operating system designed just to run containers. I compared it to LibreELEC, which is a purpose-built O/S that I use to run Kodi, and while it was explained to me that I was oversimplifying things a bit, it was otherwise a good analogy.

While it is, of course, being used withing AWS, it is a 100% open source project and you can get the code on Github, and the hope is that others will find it valuable and will get involved with the community. If this is something you’re into, stop by the booth and say “hi”.

Speaking of stopping by the booth, we do have some tasty sodas and Bottlerocket branded bottle openers, but the big giveaway is an awesome 3D printer. Get a raffle ticket and stop by the booth at 1:30pm on Sunday for the drawing (you must be present to win).

AWS employees are not eligible to participate. (sniff)

I am back at the Southern California Linux Expo (SCaLE) for the first time in many years, and I was surprised at how happy this makes me. It is always a well run conference and it tends to bring a lot of people I like together in one place, which means I get to meet a lot more people to like as well.

The main SCaLE sessions occur over the weekend, but there are a lot of cool things that happen in the days before. For Thursday, AWS sponsored Cloud Native Builder Day to showcase some of the amazing open source technologies one can use to solve a number of challenges, and I was eager to learn about them.

But before that I needed to get registered. The first step was to show proof of vaccination. While I am thankful that we can have these events, COVID is still a thing and the organizers are doing all they can to mitigate the risk to the conference attendees. Since I’m an old I’ve had two shots and two boosters but the darn thing keeps mutating.

Once past that I headed upstairs where I could use the self check-in kiosks. It was pretty simple to sign in and get my badge printed, and then it was just a short trip down the hall to pick up the conference “swag bag” which included the badge holder and lanyard.

The only change I would make to the process is that once you printed your badge, you should really hit the “close window” button on the screen, as there is a “back” button that could allow the next person who registers to see your name and e-mail. No biggie, but the security nerd in me always thinks about these things.

The conference spans two floors. The Exhibit Hall with the sponsor booths is on the ground floor behind registration (it is technically in the Plaza Ballroom so I just followed the signs for “ballrooms”) while the sessions are on the second floor along with registration. AWS is going to have a pretty cool booth this year.

As an AWS employee I guess I should say that we always have a cool booth (grin) but I especially like the idea behind this one, despite the fact that we were unable to get a mounted deer head (seriously). It’s booth numbers 300, 302 and 304 if you want to swing by, and for those of you who couldn’t make it I’ll be sure to post about it later.

Cloud Native Builder Day showcased three different open source projects, the first one being Triggermesh. This was presented by Jeff Naef who I immediately liked as he was the first to notice that my mask is made by K&N, a company known for their high-end automotive airflow products. He loves performance automobiles as well as open source (he was wearing a Snap-On tools hat) so I knew we would get along.

In dealing with cloud native technologies, a lot of the workflow is event driven. Triggermesh lets you seamlessly link together sources and targets for events, normalizing and enriching them along the way. While it does support the ability to create functions using code (in a variety of languages) a lot of the implementation can be done just through configuration.

In one example the data was encoded in base64, and a person asked if Triggermesh could render that in clear text. Jeff was like, sure, and he bravely set out to implement that as we watched. He got really close, but in any case deserves kudos for the attempt, especially considering he was holding a microphone with one hand the entire time.

The next speaker was Zoe Steinkamp from InfluxDB. I first met Zoe at the Open Source Summit in Austin and she is one of my favorite new acquaintances I’ve met through my job at AWS.

Now full disclosure: I missed the first half of her presentation.

SCaLE has done something delightful with the schedule, which is allowing 30 minutes between talks. I’ve talked about this before but this lets speakers switch out without the usual urgency, allows more time for attendees to interact with the speaker after the talk, and improves the hallway track.

I thought I had enough time to grab lunch, which was In-N-Out that Spot had brought for me. We don’t have In-N-Out in North Carolina so I rarely pass up a chance to get it, and I figured I could be back in time. I was wrong. But I did slip into the back of the room which is why this picture isn’t as close as the others.

I used to work on an open source project that relied heavily on time series data, so I’m a bit of a time series data geek. Every time I see a presentation on InfluxDB I learn more things to like about it. This time I found out that it is possible to get started with it without being a programmer. A lot of people in the data science field aren’t coders, but they can send their data to InfluxDB pretty easily. The folks at Influx have created InfluxDB University as a free resource to get the most out of their solution, and while I haven’t gone through it yet it looks really comprehensive.

The final presenter was Matt Overstreet from Datastax. Datastax focuses on providing solutions around the Apache Cassandra project, which is a distributed “NoSQL” database.

When most people hear the word “database” they think of relational databases. This is a data structure usually based on “rows” of data made up of “fields” and indexed by a primary key. One then uses something like the Structured Query Language (SQL) to retrieve values from those fields. This is all well and good but it tends to be extremely monolithic, which doesn’t work well in today’s distributed cloud environment.

Think about it. In a datacenter you might have sub-millisecond latency, so a query can be returned quickly. Move that datacenter across the country, and now it your latency is, say, 100ms. Move that to the other side of the world and, well, you get the picture. Now if you only have a few queries that might be okay, but when you consider thousands and then millions of queries, the response time of your application is going to take a hit.

Cassandra allows you to distribute that data both within a datacenter (for reliability) and also regionally. You can then put your data near your customers, improving their experience.

I was already sold on Cassandra (we used it at OpenNMS) but what I learned from this presentation was the wonderfulness that is “k8ssandra” (kate-sandra). This is Cassandra but running in Kubernetes. If you have ever had to extend and expand a Cassandra cluster, you know that while it isn’t super difficult there are a number of gotchas that can cause problems. What if you could automate it? Matt showed us an example that let him spin up (and tear down) an 800 node cluster in minutes.

Cool, huh?

The first day of SCaLE 19x was a blast, and I am eager to see what the rest of the week brings.

I always feel a little sad on the last day of any conference, and Open Source Summit was no different. It seems like the week went by too fast.

With the Sponsor Showcase closing on Thursday, attendance at the Friday keynotes was light, but those of us that showed up got to hear some pretty cool presentations.

The first one was from Rachel Rose, who supervises R&D at Industrial Light and Magic. As a fanboy of ILM I was very eager to hear what she had to say, and she didn’t disappoint. (sorry about the unflattering picture but I took three and they were all bad)

In the past a lot of special effects that combine computer generated imagery (CGI) and live action are created separately. The live action actors perform in front of a green screen and the CGI backgrounds are added later. Technology has advanced to the point that the cutting edge now involves live action sets that are surrounded by an enormous, curved LED screens, and the backgrounds are projected as the actors perform.

This presents a number of challenges as the backgrounds may need to change as the camera moves, but it provides a much better experience for the actors and the audience.

The tie-in to open source is that a lot of the libraries used the creation of these effects are now open. In fact, the Academy of Motion Picture Arts and Sciences (the people responsible for the Oscars) along with the Linux Foundation have sponsored the Academy Software Foundation (ASWF) to act as a steward for the “content creation industry’s open source software base”. The projects under the ASWF fall into one of two tiers: Adopted and Incubation. Currently there are four projects that are mature enough to be adopted and several more in the incubation stage.

A lot of this was so specific to the industry that it went over my head, but I could understand the OpenEXR project, which provides a reference implementation of the EXR file format for storing high quality images.

She then went on to talk about Stagecraft, which is the name of the ILM platform for producing content. I would love to be able to visit one day. It would be so cool to see a feature being made with the CGI, sets and actors all integrated.

The next speaker was Vini Jaiswal, Developer Advocate for Databricks. I had seen a cool Databricks presentation back on Day 2 and the first part was similar, but Jaiswal skipped the in-depth technical details and focused more on features and adoption. A rather large number of companies are using the Delta Lake technology as a way to apply business intelligence to data lakes, and as the need to analyze normally unstructured data becomes more important, I expect to see even more organizations adopt it.

The third presentation was a video by Dmitry Vinnik of Meta on measuring open source project health.

Begin rant.

To be honest I was a little unhappy to see a video as a keynote. It was the only one for the entire week and I have to admit I kind of tuned it out. It wasn’t even novel, as he has given it at least twice before. The video we were shown is available on Youtube from a conference earlier in the month and he posted another one dated June 24th from the Python Web Conference (while it has a different splash screen it looks to be the same presentation).

Look, I’ve given the same talk multiple times at different conferences, so I get it. But to me keynotes are special and should be unique. I was insulted that I bothered to show up in person, wear a mask, get my temperature checked each day, and I expected something better than a video I could have watched at home.

Note: Rachel Rose played a video as part of her presentation and that’s totally cool, as she didn’t “phone in” the rest of it.

Okay, end rant.

The next two presenters were very inspiring young people, and it was nice to have them included as part of the program.

The first speaker was Alena Analeigh, an amazing young woman who, among other achievements, has been accepted to medical school at age 13 (note that in trying to find a reference for that I came up blank, except for her twitter bio, so if you have one please let me know and I can update this post).

Med school is just one of her achievements. She also founded The Brown STEM Girls as an organization to get more women of color interested in science, technology, engineering and math. She stated that while men make up 52% of the workforce, they represent 76% of people employed in STEM fields.

My love of such things was fostered at an early age, and programs like hers are a great step to encourage young women of color to get interested in and eventually pursue careers in STEM.

While she seemed a little nervous and tentative while presenting, the final speaker of the morning was the exact opposite. At 11 years old, I could listen to Orion Jean speak for hours.

Orion also has a number of accolades, including Time Magazine’s “Kid of the Year“. He got his start as the winner of a speech contest sponsored by Think Kindness, and since then has started the Race to Kindness (“a race where everybody wins”) to spread kindness around the world.

To help inspire acts of kindness he uses the acronym K.I.N.D.:

• Keep Your Eyes Open: Look for opportunities to be kind to others. One example he used is one I actually practice. If you are in line to check out at the store, and you see a person with a lot less items than you, while not offer to let them check out first?
• Include Others: No one can effect change alone. Get others involved.
• Nothing Is Too Small: One thing that keeps us from spreading kindness is that we can try to think too big. Even small acts of kindness can have a huge impact.
• Do Something About It: Take action. Nothing can change if we do nothing.

After the keynotes I had to focus on some work stuff that I had let languish for the week, so I didn’t make it to any of the presentations, but overall I was happy with my first conference in three years.

There were a few people that attended who tested positive for COVID, so I plan to take some precautions when I get home and hope that the steps the Linux Foundation took to mitigate infection worked. So far I’ve tested negative twice, and I’ll probably take another test on Monday.

My next conference will be SCaLE in Los Angeles at the end of July, and I plan to be in Dublin, Ireland for Open Source Summit – Europe. If you are comfortable getting out and about I hope to see you there.

The word for Day 2 of the Open Source Summit is SBOM.

When I first heard the term my thought was that someone had spoken a particular profanity at an inappropriate time, but SBOM in this context means “Software Bill of Materials”. Open source is so prevalent these days that it is probably included in a lot of the software you use and you may not be aware of it, so when an issue is discovered such as Log4shell it can be hard to determine what software is affected. The idea of asking all vendors (both software-only and software running on devices) to provide an SBOM is a first step to being able to audit this software.

It isn’t as easy as you might think. The OpenNMS project I was involved with used over a hundred different open source libraries. I know because I once did a license audit to make sure everything being used had compatible licenses. I also have used Black Duck Software (now Synopsys) to generate a list of included software, and it looks like they now offer SBOM support as well, but I get ahead of myself.

Note that Synopsys is here in the Sponsor Showcase but when I stopped by the booth no one was there.

Getting back to the conference, the second morning keynotes were more sparsely attended than yesterday, but the room was far from empty. The opening remarks were given by Mike Dolan, SVP and GM of Projects at the Linux Foundation, and he was a last minute replacement for Jim Zemlin, who was not feeling well.

Included in the usual housekeeping announcements was a short “in memoriam” for Shubhra Kar, the Linux Foundation CTO who passed away unexpectedly this year.

Dolan also mentioned that the Software Package Data eXchange (SPDX) open standard used for creating SBOMs had turned 10 years old (and it looks like it will hit 11 in August). This was relevant because with applications of any complexity including hundreds if not thousands of open source software projects, there had to be some formal way of listing them for analysis in an SBOM, and most default to SPDX.

The next speaker was Hilary Carter who is in charge of research for the Linux Foundation.

She spoke on the work the Linux Foundation is doing to measure the worldwide impact of open source. As part of that she mentioned that there is a huge demand for open source talent in the market place, but there are also policy barriers for employees of many companies to contribute to open source. She also brought up SBOMs as a way to determine how widespread open source use is in modern applications.

Since diversity has been a theme at this conference I wanted to address a pet peeve of mine. This is a slide from Carter’s presentation and it uses a stylized Mercator projection to show the world. I just think it is about time we stop using this projection, as the continent highlighted, Africa, is actually much, much larger in proportion to the other continents than is shown on this map. As an alternative I would suggest the Gall-Peters projection.

To further digress, I asked my friend Ben to run “stylized Gall-Peters projection” through Midjourney but I didn’t feel comfortable posting any of the results (grin).

Anyway, enough of that. The next presenter was Kevin Jakel, who founded Unified Patents.

The goal of Unified Patents is to protect open source from patent trolls. Patent trolls are usually “non-practicing entities” who own a lot of patents but exist to extract revenue from companies they believe are infringing upon them versus building products. Quite frequently it is cheaper to settle than pursue legal action against these entities and this just encourages more actions on the part of the trolls.

The strategy to combat this is described as “Detect, Disrupt and Deter”. For a troll, the most desired patents are ones that are broad, as this means more companies can be pursued. However, overly broad patents are also subject to review, and if the Patent and Trademark Office is convinced a patent isn’t specific enough it can invalidate it, destroying the revenue stream for the patent troll.

I’m on the fence over software patents in general. I mean, let’s say a company could create a piece of software that exactly modeled the human body and how a particular drug would interact with it, I think that deserves some protection. But I don’t think that anyone owns the idea of, say, “swipe left to unlock”. Also it seems like software rights could be protected by copyright, but then again IANAL (one source for more information on this is Patent Absurdity)

The next person on stage was Amir Montazery, of the Open Source Technology Improvement Fund. The mission of the OSTIF is to help secure open source software. They do this through both audits and fundraising to provide the resources to open source projects to make sure their software is secure as possible.

Jennings Aske, of New York-Presbyterian Hospital spoke next. I have worked a bit with technology in healthcare and as he pointed out there are a lot of network connected devices used in medicine today, from the devices that dispense drugs to the hospital beds themselves. Many of those do not have robust security (and note that these are proprietary devices). Since a hack or other breach could literally be a life and death situation, steps are being taken to mitigate this.

I enjoyed this talk mainly because it was from the point of view of a consumer of software. As customers are what drive software revenues, they stand the best chance in getting vendors to provide SBOMs, along with government entities such as the National Telecommunications and Information Administration (NTIA). The NTIA has launched an effort called Software Component Transparency to help with this, and Jennings introduced a project his organization sponsors called DaggerBoard that is designed to scan SBOMs to look for vulnerabilities.

The next keynote was from Arun Gupta of Intel. His talk focused on building stronger communities and how Intel was working to build healthy, open ecosystems. He pointed out that open source is based largely on trust, which is an idea I’ve promoted since I got involved in FOSS. Trust is something that can’t be bought and must be earned, and it is cool to see large companies like Intel working toward it.

The final presenter was Melissa Smolensky from Gitlab who based her presentation around a “love letter to open source”. It was cute. I too have a strong emotional connection to my involvement in free and open source software that I don’t get anywhere else in my professional life, at least to the same degree.

I did get to spend some time near the AWS booth today, and after chatting at length with the FreeRTOS folks I happened to be nearby when Chris Short did a presentation on GitOps.

In much the same way that Apple inspired a whole generation of Internet-focused products to put an “i” in front of their name, DevOps has spawned all kinds of “Ops” such as AIOps and MLOps and now GitOps. The idea of DevOps was built around creating processes to more closely tie software development to software operation and deployment, and key to this was configuration management software such as Puppet and Ansible. Instead of having to manage configuration files per instance, one could store them centrally and use agents to deploy them into the environment. This central repository allows for a high degree of control and versioning.

It is hard to think of a better tool for versioning than git, and thus GitOps was born. Software developed using GitOps is controlled by configuration files (usually in YAML) and using git to make changes.

While I am not an expert on GitOps by any means, suppose your application used a configuration file to determine the various clusters to create. To generate a new cluster you would just edit the file in your local copy of the repo, git commit and git push.

You application would then use something like Flux (not to be confused with the Flux query language from InfluxData) to note that a change has occurred and then do a git pull which would then cause the change to be applied.

Pretty cool, huh? A lot of people are familiar with git so it makes the DevOps learning curve a lot less steep. It also allows for the configuration of multiple repositories so you can control, say, access to secrets differently than the main application configuration.

Also while I was in the booth I got this picture of two Titans of Open Source, Spot Callaway and Brian Proffitt. Oh yeah.

My final session of the day was given by Kelly O’Malley of Databricks on Delta Lake.

Now as someone who has given a lot of talks, I try to be respectful of the presenter and with the exception of the occasional picture and taking notes I try to stay off my phone. I apologized to her afterward as I was spending a lot of time looking up terms with which I was unfamiliar, such as “ACID” and “parquet“.

Delta Lake is an open source project to create a “Lakehouse”. The term is derived from a combination of “Data Warehouse” and “Data Lake“.

Data warehouses have been around for a very long time (in one of my first jobs I worked for a VAR that built hardware solutions for storing large data warehouses) and the idea was to bring together large amounts of operational data into one place so that “business intelligence” (BI) could be applied to help make decisions concerning the particular organization. Typically this data has been very structured, such as numeric or text data.

But people started figuring out that a lot of data, such as images, needed to be stored in more of a raw format. This form of raw data didn’t lend itself well to the usual BI analysis techniques.

Enter Delta Lake. Based on Apache Spark, it attempts to make data lakes more manageable and to make them as useful as data warehouses. I’m eager to find the time to learn more about this. When I was at OpenNMS we did a proof of concept about using Apache Spark to perform anomaly detection and it worked really well, so I think it is perfectly matched to make data lakes more useful.

My day ended at an internal event sponsored by Nithya Ruff, who in addition to being the chairperson of the Linux Foundation is also the head of the AWS OSPO. I made a number of new friends (and also got to meet Amir Montazery from the morning keynotes in person) but ended up calling it an early night because I was just beat. Eager to be fresh for the next day of the conference.

The main activities for the Open Source Summit kicked off on Tuesday with several keynote sessions. The common theme was community and security, including the Open Source Security Foundation (OpenSSF).

The focus on security doesn’t surprise me. I was reminded of this xkcd comic when the Log4shell exploit hit.

At the time I was consulting for a bank and I called the SVP and said “hey, we really need to get ahead of this” and he was like “oh, yeah, I was invited to a security video call a short while ago” and I was like “take the call”.

I managed to squeeze into the ballroom just before the talks started, and I was happy to see the room was packed, and would end up with a number of people standing in the back and around the edges.

The conference was opened by Robin Bender Ginn, Executive Director of the OpenJS Foundation.

After going over the schedule and other housekeeping topics, she mentioned that in recognition of Pride Month the conference was matching donations to the Transgender Education Network of Texas (TENT) as well as Equality Texas, up to $10,000.

In that vein the first person to speak was Aeva Black, and they talked about how diversity can increase productivity in communities, specifically open source communities, by bringing in different viewpoints and experiences. It was very well received, with many people giving a standing ovation at its conclusion.

The next speaker was Eric Brewer from Google (a platinum sponsor) and his talk focused on how to improve the robustness and security of open source (and he joked about having to follow Black with such a change of topic). Free software is exactly that, free and “as is”. So when something like Log4shell happens that impacts a huge amount of infrastructure, there is really no one who has an implicit obligation to rectify the issue. That doesn’t prevent people from trying to force someone to fix things, as this infamous letter to Daniel Stenberg demonstrates.

Brewer suggests that we work on creating open source “curators” who can provide commercial support for open source projects. In some cases they could be the maintainer, but it is not necessary. When I was at OpenNMS our support offerings provided some of this indemnification along with service levels for fixing issues, but of course that came at a cost. I think it is going to take some time for people to realize that free software does not mean a free solution, but this idea of curators is a good start.

I got the feeling that the next presentation was one reason the hall was so packed as Linus Torvalds and Dirk Hohndel took the stage. Linus will be the first to admit that he doesn’t like public speaking, but I found that this format, where Dirk asked him questions and he responded, worked well. Linus, who is, well, not known for suffering fools gladly, admitted and apologized for his penchant for being rather sharp in his criticism, and when Dirk asked if he was going to be nicer in the future Linus said, no, he probably wouldn’t so he wanted to proactively apologize. That made me chuckle.

This was followed by a security-focused presentation by Todd Moore from IBM, another platinum sponsor. He also addressed trying to improve open source security but took an angle more aimed at government involvement. Digital infrastructure is infrastructure, much like bridges, roads, clean water, etc., and there should be some way for governments to fund and sponsor open source development.

The final keynote for today was a discussion with Amy Gilliland who is the President of General Dynamics Information Technology (GDIT). In a past life I worked quite a bit with GDIT (and you have to admit, that can be a pretty appropriate acronym at times) and it is nice to see a company that is so associated with more secretive aspects of government contracting focusing on open source solutions.

After the keynotes I visited the Sponsor Hall to see the AWS booth. It was pretty cool. As a diamond sponsor it is right in front as you enter.

There were people from a number of the open source teams at AWS available to do presentations, including FreeRTOS and OpenSearch.

I don’t have booth duty this conference so I decided to wander around. I thought it was laid out well and it was interesting to see the variety of companies with booths. I did take some time to chat with the folks at Mattermost.

While I’m a user of both Discord and Slack, I really, really like Mattermost. It is open source and provides a lot of the same functionality as Slack, and you can also host it yourself which is what the OpenNMS Project does. If you don’t want to go to the trouble of installing and maintaining your own instance, you can get the cloud version from Mattermost, and I learned that as of version 7 there is a free tier available so there is nothing preventing you from checking it out.

I did take a short break from the conference to grab lunch with my friend William Hurley (whurley). It had been at least three years since we’d seen each other face to face and, thinking back, I was surprised at the number of topics we managed to cover in our short time together. He is an amazing technologist currently working to disrupt, and in many ways found, commercial quantum computing through his company StrangeWorks. He also made me aware of Amazon Braket, which lets those of us who aren’t whurley to access quantum computing services. I’m eager to check it out as it is an area that really interests me.

After lunch I was eager to see a presentation on InfluxDB by Zoe Steinkamp.

Time series data collection and storage was a focus of mine when I was involved in monitoring, and Influx is working to make flexible solutions using open source. Steinkamp’s presentation was on combining data collection at the edge with backend storage and processing in the cloud. Influx had a working example of a device that would monitor the conditions of a plant (she’s an avid gardener) such as temperature and moisture, and this data was collected locally and then forwarded to the cloud. They have a new technology called Edge Data Replication designed to make the whole process much more robust.

I was excited to learn about their query language. Many time series solutions focus so much on obtaining and storing the data and not enough on making that data useful, which to me seems to be the whole point. I’m eager to play with it as soon as I can.

One thing that bothered me was that the hotel decided to have the windows washed in the middle of the presentation.

Steinkamp did a great job of soldiering through the noise and not letting it phase her.

The evening event was held at Stubbs restaurant, which is also a music venue.

I’ve been a fan of Stubbs barbecue sauce for years so it was cool to go to the restaurant that bears his name, even though the Austin location was opened in 1996, a year after Christopher B. Stubblefield died.

It was a nice end to a busy day, and I look forward to Day 2.

Monday was a travel day, but it was notable as it was the first time I have been in an airport since August. I fly out of RDU, and the biggest change was that they now have the “Star Trek” x-ray machines to scan carry-on luggage. While I was panicked for a second when I downloaded my boarding pass and didn’t see the TSA Precheck logo, I was able to get that sorted out so going through security was pretty easy.

The restrictions on masks for air travel have been lifted, but I wore mine along with about 10% of the other travelers. Even though I’ve had four shots and a breakthrough case of COVID I do interact with a lot of older people and since I’ll be around the most people in years at the Open Source Summit I figured I’d wear mine throughout the trip.

And while it isn’t N95, being a car nut I tried out these masks from K&N Engineering, who are known for high end air filtration for performance vehicles, and you almost don’t realize you are wearing a mask.

Anyway, I made my way to the Admiral’s Club and was pleasantly surprised to see it wasn’t very crowded. It was nice to have the membership (it comes with my credit card) as my flight to Charlotte was delayed over 90 minutes. I wasn’t too worried since I had a long layover before heading to Austin, so I was a lot less stressed than many of my fellow travelers.

The flight to Austin left on time and landed early, but we got hit with the curse in that our gate wasn’t available, so we ended up on the tarmac for 45 minutes, getting in 30 minutes late.

Not that I’m complaining. Seriously, according to my handy the trip from my home to Austin by car is 19 hours. From the moment I left my home until we landed was more like 8 hours, and most of that was enjoyable. I always have to remind myself of this wonderful clip by Louis CK which kind of sums up the amazing world in which we live where every time we fly we should be saying to ourselves “I’m in a chair in the sky!”

I checked in at the hotel and then we headed back out in our rented minivan to get the last member of our team, and then we drove about 45 minutes outside of Austin to this barbecue joint called Salt Lick in Driftwood Texas. It was wonderful and I was told we owed this experience to a recommendation years ago from Mark Hinkle, so thanks Mark!

You can’t really tell a good barbecue restaurant by its looks, although shabbier tends to be better, but more by the smell. When you get out of your vehicle your nose is so assaulted with the most wonderful smell you might be drawn to the entrance so quickly that you miss the TARDIS.

We sat at a big picnic table and ordered family style, which was all you could eat meat, slaw, baked beans, bread, pickles and potato salad. I was in such a food coma by the end that I forgot to take a picture of the cobbler.

I tried not to fall asleep on the ride back to Austin (I wasn’t driving) but it was a great start to what I hope is a wonderful week.

Next week I’ll be attending my first conference in nearly three years. My last one turned out to be the very last OSCON back in 2019. Soon after that I was in a bad car accident that laid me up for many months and then COVID happened.

I am both eager and anxious. Even having four vaccine shots and one breakthrough case I still feel a little exposed around large groups of people, but the precautions outlined in the “Health and Safety” section of the conference website are pretty robust and I am eager to see folks face-to-face (or mask-to-mask) once again.

The Linux Foundation’s Open Source Summit used to be known as Linuxcon and now it is an umbrella title for a number of conferences around open source, all of which look cool. My new employer, AWS, is a platinum sponsor and will also have a booth (I am not on booth duty this trip but I’ll be around). I am looking forward to getting to meet in person many of my teammates who I’ve only seen via video, old friends I haven’t seen in years, and to making a bunch of new ones.

Of course, we would have to have a conference in Austin during a heat wave. I was thinking about never leaving the conference venue but then I remembered … barbecue.

If you are going and would like to say “hi” drop me a note on Twitter or LinkedIn or send an e-mail to tarus at tarus dot io.

For several years now the OSMC has been extended by one day in the form of a “hackathon”. As I do not consider myself a developer I usually skip this day, but since I wanted to spend more time with Ronny Trommer and to explore the OpenNMS MQTT plugin, I decided to attend this year.

I’m glad I did, especially because the table where we sat was also home to Dave Kempe, and he brought Tim Tams from Australia:

Yum.

You can find them in the US on occasion, but they aren’t as good.

I have been hearing about MQTT for several years now. According to Wikipedia, MQTT (Message Queuing Telemetry Transport) is a messaging protocol designed for connections with remote locations where a “small code footprint” is required or the network bandwidth is limited, thus making it useful for IoT devices.

Dr. Craig Gallen has been working on a plugin to allow OpenNMS to consume MQTT messages, and I was eager to try it out. First, we needed a MQTT broker.

I found that the OpenHAB project supports an MQTT broker called Mosquitto, so we decided to go with that. This immediately created a discussion about the differences between OpenHAB and Home Assistant, the latter being a favorite of Dave. They looked comparable, but we decided to stick with OpenHAB because a) I already had an instance installed on a Raspberry Pi, and b) it is written in Java, which is probably why others prefer Home Assistant.

Ronny worked on getting the MQTT plugin installed while I created a dummy sensor in OpenHAB called “Gas”.

This involved creating a “sitemap” in /etc/openhab2:

sitemap opennms label="My home automation" {
    Frame label="Date" {
        Text item=Date
    }
    Frame label="Gas" {
        Text item=mqtt_kitchen_gas icon="gas"
    }
}

and then an item that we could manipulate with MQTT:

Number mqtt_kitchen_gas "Gas Level [%.1f]" {mqtt="<[mosquitto:Home/Floor1/Kitchen/Gas_Sensor:state:default]"}

To install the MQTT plugin:

• Create folder deploy in ${OPENNMS_HOME}
• Download the KAR file or compile it from source and drop it to the deploy folder.
• Create a mqtt-interface-config.xml file

Ronny added the following to the configuration to connect to our Mosquitto broker on OpenHAB:

<mqttclients>
  <client clientinstanceid="client1">
    <brokerurl>tcp://172.20.11.8:1883</brokerurl>
    <clientid>opennms</clientid>
   <connectionretryinterval>3000</connectionretryinterval>
    <clientconnectionmaxwait>20000</clientconnectionmaxwait>
    <topiclist>
      <topic qos="0" topic="iot/#">
    </topic>
    <username>openhabian</username>
    <password>openhabian</password>
    </client>
</mqttClients>

Now that we had a connection between our OpenHAB Mosquitto broker and OpenNMS, we could try to send information. The MQTT plugin handles both event information and data collection. To test both we used the mosquitto_pub command on the CLI.

For an event one can use something like this:

#/bin/bash
mosquitto_pub -u openhabian --pw openhabian -t "iot/timtam" -m "{ \"name\": \"6114163\",  \"sensordatavalues\": [ { \"value_type\": \"Gas\", \"value\": \"$RANDOM\"  } ] }"

On the OpenNMS side you need to configure the MQTT plugin to look for it:

<messageEventParsers>
  <messageEventParser foreignSource="$topicLevels[5]" payloadType="JSON" compression="UNCOMPRESSED">
    <subscriptionTopics>
      <topic>iot/timtam/event/kitchen/mysensor/doorlock</topic>
    </subscriptionTopics>

    <xml-groups xmlns="http://xmlns.opennms.org/xsd/config/xml-datacollection">
      <xml-group name="timtam-mqtt-lab" resource-type="sensors" resource-xpath="/" key-xpath="@name">
        <xml-object name="instanceResourceID" type="string" xpath="@name"/>
        <xml-object name="gas" type="gauge" xpath="sensordatavalues[@value_type="Gas"]/value"/>
      </xml-group>
    </xml-groups>
    <ueiRoot>uei.opennms.org/plugin/MqttReceiver/timtam/kitchen
  </messageEventParser>
</messageEventParsers>

Note how Ronny worked our Tim Tam obsession into the configuration.

To make this useful, you would want to configure an event definition for the event with the Unique Event Identifier (UEI) of uei.opennms.org/plugin/MqttReceiver/timtam/kitchen:

<events xmlns="http://xmlns.opennms.org/xsd/eventconf">
  <event>
    <uei>uei.opennms.org/plugin/MqttReceiver/timtam/kitchen</uei>
    <event-label>MQTT: Timtam kitchen lab event</event-label>
    <descr>This is our Timtam kitchen lab event</descr>
    <logmsg dest="logndisplay">
      All the parameters: %parm[all]%
    </logmsg>
    <severity>Normal</severity>
    <alarm-data reduction-key="%uei%:%dpname%:%nodeid%:%interface%:%service%" alarm-type="1" auto-clean="false"/>
  </event>
</events>

Once we had that working, the next step was to use the MQTT plugin to collect performance data from the messages. We used this script:

#!/bin/bash
while [ true ]
do
mosquitto_pub -u openhabian --pw openhabian -t "Home/Floor1/Kitchen/Gas_Sensor" -m "{ \"name\": \"6114163\",  \"sensordatavalues\": [ { \"value_type\": \"Gas\", \"value\": \"$RANDOM\"  } ] }"
sleep 10
done

This will create a message including a random number every ten seconds.

To have OpenNMS look for it, the MQTT configuration is:

<messageDataParsers>
  <messageDataParser foreignSource="$topicLevels[5]" payloadType="JSON" compression="UNCOMPRESSED">
    <subscriptionTopics>
      <topic>iot/timtam/metric/kitchen/mysensor/gas</topic>
    </subscriptionTopics>
    <xml-groups xmlns="http://xmlns.opennms.org/xsd/config/xml-datacollection">
      <xml-group name="timtam-kitchen-sensor" resource-type="sensors" resource-xpath="/" key-xpath="@name">
        <xml-object name="instanceResourceID" type="string" xpath="@name" />
        <xml-object name="gas" type="gauge" xpath="sensordatavalues[@value_type="Gas"]/value"/>
      </xml-group>
    </xml-groups>
    <xmlRrd step="10">
      <rra>RRA:AVERAGE:0.5:1:20160</rra>
      <rra>RRA:AVERAGE:0.5:12:14880</rra>
      <rra>RRA:AVERAGE:0.5:288:3660</rra>
      <rra>RRA:MAX:0.5:288:3660</rra>
      <rra>RRA:MIN:0.5:288:3660</rra>
    </xmlRrd>
  </messageDataParser>
</messageDataParsers>

This will store the values in an RRD file which can then be graphed within OpenNMS or through Grafana with the Helm plugin.

It was pretty straightforward to get the OpenNMS MQTT plugin working. While I’ve focused mainly on what was accomplished, it was a lot of fun chatting with others at our table and in the room. As usual, Netways did a great job with organization and I think everyone had fun.

Plus, I got to be reminded of all the amazing stuff being done by the OpenNMS team, and how the view is great up here while standing on the shoulders of giants like Ronny and Craig.

Despite how long the Tuesday night festivities lasted, quite a few people managed to make the first presentation on Wednesday morning. I’m old so I had gone to bed fairly early and was able to see “Make IT Monitoring Ready for Cloud-native Systems” bright and early.

This presentation focused on a project called RealOpInsight. This seems to be a sort of “Manager of Managers” for multiple monitoring applications, and I didn’t really see a “cloud-native” focus in the presentation. It is open-source so if you find yourself running many instances of disparate monitoring platforms you may find RealOpInsight useful.

This was followed by a presentation from Uber.

One can imagine the number of metrics an organization like Uber collects (and I did refrain myself from making snarky comments like “what database do you use to track celebrities?” and “where do you count the number of assaults by Uber drivers?”). Rob Skillington seemed pretty cool and I didn’t want to put him on the spot.

Uber used to use Cassandra, which is a storage option for OpenNMS, but they found when they hit around 80,000 metrics per second the system couldn’t keep up (one of the largest OpenNMS deployments is 20,000 metrics/sec so 80K is a lot). Their answer was to create a new storage system called M3DB. While it seems pretty impressive, I did ask some questions about how mature it was because at OpenNMS we are always looking out for ways to make things easier for our users, and Rob admitted that while it works well for Uber it needs some work to be generally useful, which is why they open-sourced it. We’ll keep an eye on it.

The next time slot was the “German only” one I mentioned in my last post, so I engaged in the hallway track until lunch.

It was lovely to see Rihards Olups again. We met at the first OSMC I attended when he was part of the “Latvian Army” at Zabbix. He gave an entertaining talk on dealing with the alerts from your monitoring system, and he ended with the tag line “Make Alerts Meaningful Again (MAMA)”. Seems like a perfect slogan for a ball cap, preferably in red.

Another delightful human being I got to see was Dave Kempe, who came all the way from Sydney. While we had met at a prior OSMC, this conference we ended up spending a lot more time together (he was in the Prometheus training as well as the Thursday Hackathon). He gave a talk on being a monitoring consultant, and it was interesting to compare his experiences with my own (they were similar).

For most people the conference ended on Wednesday. I said goodbye to people like Peter Eckel and looked forward to the next OSMC so I could see them again.

Speaking of the next OSMC, we are going to be doing OpenNMS training on that first day, November 4th, so save the date. It is the least we could do since they went to the trouble to advertise OpenNMS Horizon® on all their posters (grin).

Ronny and I were hanging around for the Hackathon on Thursday, and for those attendees there was a nice dinner at a local restaurant called Tapasitos. It was fun to spend more time with the OSMC gang and to get ready for our last day at the conference.