NOTE: When I initially wrote this article I used “they/them” pronouns when referring to Hazel Weakly instead of the preferred “she/her” pronouns shared in her speaker notes. People have pointed out that this was a microaggression. This was not my intent, and I recognize I made a mistake and I’m sorry. I corrected her pronouns and promise to do better in the future. Thanks to everyone who informed me of my error.
The 22nd edition of the Southern California Linux Expo (SCaLE 22x) is now over. Once again it was a great experience and is why SCaLE is my favorite open source event of the year.
TL;DR; Hats off to the Ilan, Hannah, Bala and the rest of the SCaLE team for another amazing four days of open source goodness. This all-volunteer event puts on a conference that provides fun, knowledge and a great sense of community that can’t be duplicated.
I landed at LAX Thursday afternoon and slowly made my way through LA traffic to Pasadena. As I was walking into the hotel I heard someone call my name, and my friend Mark Hinkle was sitting at the bar. After dropping off my bags at the room I joined him, and was introduced to two other cool people, Greg and Craig.
We caught up and chatted about a bunch of things, then I had to leave to attend an event sponsored by Flox. Flox is heavily involved in NixOS, and they were sponsoring PlanetNix at the conference. The main presentation days at SCaLE are Saturday and Sunday, but over the years the two “lead up” days have seen more and more people in attendance. PlanetNix drove a lot of that traffic and I was eager to catch up with my friends Ron Efroni and Ross Turk.
It turns out Craig was a big Nix user so I said he could be my plus one, and we headed over to the venue. It was raining (which was a bit unusual for Southern California) but the place was packed with people. I thought it would be an open bar but when I ordered my drink I was told I needed a ticket, so we went back out to the deck and found the ticket person. We ended up meeting Itamar who had just moved to LA from Israel, and while we had a great time I never saw Ron or Ross, and I was a little disappointed they didn’t put out any pizza. The three of us headed out and got some tacos before calling it a night.
I found out later we had gone to the wrong event. Ooops. So to whoever it was that bought my drink - thanks! The Flox event was at the same restaurant just in a private room on the other side from where we were sitting.
The next morning I had work stuff to do until just before lunch, then I headed over to the convention center to get checked in. One of the main things I like about SCaLE is that I can’t walk 50 feet without running into someone I know, and it was really fun to catch up with everyone.
I had reached out to Ron and set aside some time to meet him (which is when I learned where I had gone wrong the night before). PlanetNix was in a different building from the main conference, but it was full of people and had a lot of energy.
On the way I had to walk past the theatre where the television show America’s Got Talent is filmed. I thought I should dust of my “clean five” and have a go at it, but then thought better of it (grin)
AWS is a platinum sponsor of SCaLE so I stopped by the booth to meet up with my team. One of them, Lahari, was speaking at “Upscale”, which is a series of lightning talks. Each presenter gets five minutes and 20 slides. The slides automatically advance once every 5 seconds. It is always fun to watch so I made sure to get there early enough for a good seat.
The MC for Upscale was Hannah Anderson, and she introduced the format and encouraged us to applaud should the speaker get behind on their slides or end up having to wait for the next one to pop up. There were to be nine speakers in total.
The first speaker was Scott Williams, who has created a shell based on the programming language BASIC called “qbsh” or “queue bee shell” (not “q-bash”).
I got my first computer in 1978 and the programming language was BASIC. I thought it was cute that his slides all had line numbers: 10, 20, 30, etc.
He got a big laugh with slide 180 which showed that they distributed qbsh on a 3.5 inch floppy.
The next speaker was Katherine Druckman who spoke on the various issues with realizing the value of GenAI in production, and ended by introducing us (well, at least me) to a Linux Foundation sandbox project called the “Open Platform for Enterprise AI” which exists to foster a community around AI innovation.
Mark Waite followed Katherine with a talk about common mistakes in contributing to open source projects. I enjoyed his talk mainly because his angle was “don’t make the mistakes I’ve made”. I get a lot of value out of people talking about their own personal experiences more than someone with a more theoretical “this is what you should do” take.
I am trying to figure out how to describe the next talk kindly. I didn’t pay that much attention to it because I was distracted by the graphics he choose to use.
The talk was given by Greg Kutzbach and was on building an incident response plan. On each of his slides there was a Lego-themed graphic (Lego is big at SCaLE) that was obviously created by GenAI. While much of the image looked fine, as is the case with GenAI many of the words were misspelled. I found myself looking for those mistakes more than listening to the talk. I then tried to identify the emotion I was feeling and it turned out I was insulted. I am certain this isn’t the message Greg wanted to convey but, to me, it came across as “I know my slides have errors but I don’t care enough to do anything about it”. Maybe it is because I’m an old but it would have been a more effective presentation (at least for me) if he’d left the graphics out entirely.
I guess I’m oversensitive since I’ve seen too many people look at computer and to deny reality because the computer says different (up to and including thinking a living person is dead). GenAI is only going to make this worse.
So I guess it is a good segue that the next talk, by my amazing teammate Lahari Chowtoori, was on dealing with bias in AI.
I am lucky enough to work with Lahari at AWS and she brought up a key point about these foundational models that everyone is so excited about is that the training material leads to an inherent bias in the results. I was reminded of the story from a couple of years ago where a researcher fed a pixelated picture of Barack Obama into a GenAI system and asked it to clear it up, and it returned a very “white” version of the former President. Or the graduate student at MIT who had to wear a white mask to get a facial recognition program to recognize her. There aren’t any easy answers but the first step is acknowledging the problem.
Lahari was followed by the ever entertaining Jorge Castro. His talk was on contributing to open source projects, and it was nice to hear, once again, that you don’t have to be able to contribute code in order to be a valuable member of a project.
[Note: whenever Jorge comes up one must mention that Megadeath are better than Metallica. If you ever meet him in person be sure to bring that up]
Kirk Chu was the next presenter, and he talked about how important it was to gain communication skills as a software engineer.
The next to the last talk was interactive. Adrianna Frick lead us through some mindfulness exercises, and we did some breathing and visualization tasks, starting with our feet and working our way up to our heads.
Dave McAllister gave the final talk and it was hilarious. Correlation does not imply causation but many people still treat it as if it does, so Dave went through some very funny examples. And yes, if you use spaces over tabs in your code you will get paid more.
I enjoyed the UpScale talks and hope one day to give one. Of course, when I bring this up to my friends they all laugh and say that I can’t introduce myself in five minutes, much less give a useful talk, but at least these folks gave me examples to which I can aspire.
The main conference kicked off Saturday morning with a keynote given by Solomon Hykes. Solomon is probably best known as the inventor of Docker, and he is now a founder and CEO of Dagger.io.
The “dag” in “dagger” stands for “directed acyclic graph” and it is a concept that underlies most solutions that focus on workflows (DAGs come up a lot in projects like Apache Airflow and dbt Core). One of the cool features of Dagger is that it has built-in observability, which makes it much easier to see what’s happening, and it has an interactive terminal where you can work in real-time.
Solomon did a brave thing by attempting a live demo at a conference. This is something I have since given up due to issues with things like conference WiFi, but his went well. He used Dagger to interact with an LLM to create some text about SCaLE. I’m not the biggest fan of GenAI but his demo made me want to play with it.
I got to meet up with Solomon later to show off our blue hockey jerseys. Staff at SCaLE can be identified by the black hockey jerseys they wear, customized with their name. If you have ever given a keynote at SCaLE they gift you with one, just in blue, and I get to wear mine three days out of the year (I gave the Sunday morning keynote for SCaLE 8x).
When I saw this picture I couldn’t help but see the SCaLE logo as one of those old “I’m with stupid” t-shirts but then I would have to be standing on his right instead of his left.
After the keynote I went to several sessions. The first one was by Hazel Weakly. Her talk was entitled “Source Available is Thriving but Open Source is Dying”. Since I didn’t agree with the premise I decided to sit in.
She didn’t really address the “source available is thriving” part of the title, which was the biggest issue for me. We are seeing more companies switch their licenses from an open source one to a proprietary one but I don’t think any of those companies have seen the benefits they expected, so I disagree with “thriving”. However Hazel did talk about a number of challenges facing open source that were valid, although I don’t think we are at the “dying” phase yet. As open source gets more and more ingrained in modern technology stacks, insuring the continued existence and security of open source is a key challenge. While I don’t often plug my employer, AWS, they did invest US$10 million in the Open Source Security Foundation (OpenSSF) to help with the security aspect of open source software, and there is always more we can do.
Which is a great segue into the next talk, given by the wonderful benny Vasquez. benny is the Chair of the AlmaLinux OS Foundation, and she gave an overview of the different types of open source projects out there. Some are maintained by a single person or small number of people, some are backed by corporations and others are part of foundations. Some foundations, such as the Apache Software Foundation and the Linux Foundation, act as umbrella organizations for many projects, and others are more focused such as the Rust Foundation and the Gnome Foundation.
She laughed when she included AlmaLinux with those last two as Alma is very small.
Now I am a data geek and one thing about non-profit organizations in the US is that they have to publish a yearly tax form known as a 990. Some are very transparent and you can find those forms on the project’s website, and there are other sites out there that will let you access 990s for a fee.
Recently I found that ProPublica (a news organization I support financially and encourage you to do as well) has a “Nonprofit Explorer” tool that gives you access to 990s. Most are from 2023 since the 2024 forms aren’t due yet, so this is what I learned.
For 2023 the Rust Foundation had revenue of US$2.5 million and expenses of US$3 million, with net assets of US$1.8 million.
The Gnome Foundation had revenue of US$556K, expenses of US$654K, and net assets of US$783K.
By comparison, AlmaLinux had revenue of US$228K, expenses of US$152K and net assets of US$75K.
A couple of things to note here. In most cases the key number for non-profits is the value of net assets, since that is the “fund size” managed by the non-profit. Alma is one-tenth the size of Gnome and more than one-twentieth the size of Rust.
Also note that Alma doesn’t run at a loss, which I think is cool, but it is also most likely because they can’t (grin).
Finally, think about the size of these numbers. I know software developers who make more than US$228K in salary and yet that is what the entire AlmaLinux project runs on in a given year (although benny tells me her budget this year is a whopping US$250K).
Outside of the raw revenue numbers, you can also see executive compensation, etc., which can be enlightening.
If you think 990s are fun, I’ll leave it as an exercise for the reader to lookup orgs like the Linux Foundation or Mozilla Foundation. One organization I like to use as a benchmark is the Wikimedia Foundation. They have assets over US$250 million which may surprise some folks, and I like to compare executive compensation to fund size just for giggles.
Well, enough math. I know my three readers could care less so back to SCaLE.
The first talk I went to after lunch was my own. I work with an open source project called MapLibre, and while I am not a developer it is pretty easy to use and I like talking about the MapLibre community. The room wasn’t packed but the people that showed up seemed to enjoy the presentation (and I was up against Maddog Hall in the next room over). The biggest change in the past six months (for me) was the availability of a “globe” view, which gives you the option to display a “Google Earth” type map that starts with a view of the Earth as a globe and allows you to zoom in.
The rest of the day was spent on the expo floor and in the hallway track.
It was nice to catch up with Andy Wafaa of Arm, and while my friend Megan Knight couldn’t join us she did send along a nice Arm-branded candy bar as a present after my talk.
That evening I ran into Mark again and we decided to go to In-N-Out Burger.
I try to make it to In-N-Out at least once during any trip to LA, and I get my usual: Double Double (animal style), well done fries and a strawberry shake. Several years ago they started adding calorie counts to menus in California and for this one trip I make sure to ignore them.
Sunday morning’s keynote was by Denver Gingerich who talked about the OpenWrt One router.
OpenWrt is an open source router firmware project that started back in 2004. We actually used a Linksys WRT54G as the first router for my old company because it ran OpenWrt. The OpenWrt One project built router hardware to showcase the capabilities of OpenWrt and to maximize as much as possible open source solutions. The WiFi for the conference was provided by a whole bunch of these devices spread out across the venue.
After the keynote I went back to the expo hall, where the AWS team gave away some cool Lego sets at noon. I work with a very talented group of people and they always come up with an interesting booth for conferences, which usually includes some sort of Lego giveaway.
The closing keynote, in keeping with SCaLE tradition, was given by an Internet luminary, in this case Leslie Lamport.
Lamport is best known for being one of the creators of of the documentation markup system LaTex. I also learned that he had developed TLA+, a logic for modeling the behavior of distributed systems. It is way to define all possible system behaviors to detect any cases where the system would violate desired outcomes. In other words it is a way to mathematically describe a system to insure bad things won’t happen and that good things will happen (well, eventually).
This is used at Amazon to insure its cloud offerings, which can be quite complex, behave as expected.
Leslie also got a cool keynote jersey (grin).
With that SCaLE 22x came to a close. I had a great time this year and look forward to returning in 2026.
