The thing I like most about my job is that I get to meet and work with amazing people. Recently I traveled to Helsinki, Finland, to attend the MariaDB Server Fest conference. It was a great experience and I met some very talented people, including Monty Widenius himself.

Note: The usual disclaimer that this is my personal blog and what I write here does not necessarily reflect the views of my employer, Amazon Web Services

My role at AWS is to work with open source companies and communities and to act as a liaison between them and Amazon. In thinking about important open source projects one of the first that comes to mind is MariaDB.

When I first got seriously involved in open source back in 2001, the MySQL database was an example of an open source success story. While a lot of the focus of the early days of open source was on the operating system, MySQL demonstrated that open source applications were powerful enough to compete with existing proprietary solutions. Plus, if you were building an open source application, quite often you needed a database, and MySQL provided a great option.

Many of us expected MySQL to IPO, but instead the company was bought by Sun Microsystems. That wasn’t too worrisome since Sun was a big proponent of open source, but when Sun was bought out by Oracle a couple of years later, that all changed.

On the day the acquisition was announced, Monty Widenius (the lead developer of MySQL) announced a fork of MySQL called MariaDB. In the years since then, a lot of people have replaced MySQL with MariaDB. While Oracle has continued to work on MySQL, the last major release, version 8.0, came out in April of 2018 so one must wonder how motivated they are to work on a product that competes with their main proprietary offering.

When I learned about Server Fest I decided to attend. As much as I like the ease of remote communication, sometimes nothing beats meeting face to face. I had also been to Helsinki a couple of time before and I really like the city, although I really should try to visit once in the summer time.

I flew from North Carolina to JFK and then took a Finnair flight to Finland. Helsinki is seven hours ahead of New York, so it is one of those weird trips where you leave in the night and land the following afternoon. When I travel I tend to stay at Marriott properties, but all the Marriott affiliated hotels were booked. I later learned that this was because a popular start-up conference called Slush was happening at the same time as Server Fest. Because of this there was no meeting space for rent, so the MariaDB event was being held at Monty’s house, which I thought was kind of cool.

The conference was on Thursday, November 17th, and was going to be live-streamed on YouTube. In order to better match up with the time zone in New York, it started around 3pm and ran into the night. I arrived mid-morning.

When you walk into Monty’s house, the first thing you notice is that is has a very open floorplan. Directly across from the entryway is a huge table that can probably seat about 20 people, and that’s where most folks had set up their laptops. To the right of that was a large kitchen, and to the left was an open area where the walls were lined with bookcases, and that is where lights and cameras had been set up for the livestream.

Now MariaDB is organized in two parts. There is the MariaDB corporation, which is the main commercial enterprise behind the project, and there is the MariaDB Foundation, which manages project governance and promotion. Both were represented in the day’s presenters, and I also got to meet and spend a lot of time with Kaj Arnö, who is the CEO of the Foundation.

I also got to meet the true boss of the event, Anna Widenius, Monty’s wife. As you can imagine, getting a bunch of open source geeks organized is like herding cats, but she did a great job in getting the conference underway and keeping it moving.

The event kicked off with a presentation by Manuel Arostegui, who helps maintain the MariaDB infrastructure at the Wikimedia Foundation.

He talked about “Chasing Bugs in Production”. When Wikimedia upgraded from MariaDB 10.4 to 10.6 they ran into a performance issue. His talk describes their upgrade process and how they were able to work with MariaDB to get the issue addressed. I also found it interesting that they run MariaDB on bare metal. So much of today’s IT infrastructure is based on clouds and Kubernetes that it was refreshing to see someone taking advantage of individual servers when it makes sense.

There was a bit of a hiccup with the second speaker who was supposed to join remotely, so Monty Widenius moved his presentation on query optimization in MariaDB to the second slot.

There are several methods that can be used to execute a query against a database. A good database will optimize the query method to choose the best plan to return the result in fastest time. In MariaDB 11, Monty has changed the method to use a cost-based optimizer (versus rule-based) with parameters that can be tuned by the user. This has resulted in more efficient queries and thus a better user experience.

The third presentation was by Daniel Black who joined us via Zoom from Australia (where it was 1am the next day.

His topic involved the community, and I really liked his comment that MariaDB is “dependent on its users to give MariaDB its purpose” which I thought was pretty insightful.

The next presentation was one of my favorite. It was on MindsDB and was presented by Jorge Torres.

MindsDB allows you to integrate machine learning easily into your database. In his example he used a model trained by Hugging Face to analyze text in order to detect “sentiment” – i.e. is the text positive, negative or neutral. And you access this using SQL queries.

For example, supposed you have a blog or other website where users can submit comments. MindsDB would allow you to examine those comments to detect general sentiment without having to learn an entirely new system. I thought it was pretty cool.

The fifth presentation was on “respecting the stack” or insuring continuity within MariaDB. This was lead by Soeren von Varchmin along with Kaj Arnö and Dom Taylor.

This resonated with me as it focused on building the sponsorship community within MariaDB, for both individuals and entities. MariaDB is an important piece of technology and there are a lot of stakeholders, and this talk really reinforced the idea of a “big tent” environment within the project.

For the next presentation we finally got to hear from Federico Razzoli founder of Vettabase (he was originally scheduled to go second but there was some time zone confusion) as he talked about new MariaDB features to learn “for a happy life”.

He started off with the comment that MariaDB (and open source projects in general) are very good at creating new features and not so good about documenting or advertising them. He discussed the most recent releases of MariaDB and then highlighted various new features that people should find useful.

The seventh presentation was by Sergey Petrunia and revisited the optimizer, but focused on changes made before Monty’s changes in MariaDB 11.

His talk focused on those changes made in the last year (i.e. since the last Server Fest) and it looks like a lot of progress was made to make the optimizer more consistent.

The following talk by Marko Mäkelä was on Multi-Version Concurrency Control (MVCC) and it went right over my head.

From what I can tell, the idea behind MVCC is that active databases are constantly processing transactions but there is a need to provide a consistent “view” at a given point in time, so MVCC determines which transactions are supposed to be considered committed at that point in time and which are not. This is to prevent someone who is reading from the database from being served incomplete information.

The final scheduled talk was by Andrew Hutchings who works to analyze contributions to the MariaDB project.

As transparency is key to any open source project, MariaDB publishes statistics on code contributions. The latest one I can find is through September of this year, and I was happy to see Amazon on the list of contributors to the MariaDB Server code.

Of course the majority of code commits, nearly 80% were done by the MariaDB corporation, and another 14% by the MariaDB Foundation. Amazon represented 1.42% of contributions, but Andrew pointed out to me that they came from 14 unique committers versus 8 from the Foundation. I’d love to see that involvement increase.

We did get a bonus talk by Andrei Elkin on the GTID-aware mariadb-binlog

The mariadb-binlog is a binary log containing a record of all changes to the databases, both data and structure. There is a command line tool that lets you examine this log which now supports Global Transaction IDs, making it easier to filter transactions.

After the Server Fest stream was over, we got to my favorite part of any conference – the socializing.

I did spend some time talking with Manuel Arostegui. One of my friends, Eric Evans, works at the Wikimedia Foundation focused on Cassandra. It turns out that both Manuel and Eric are in the same department. Small world.

We eventually sat down to dinner prepared by the Widenius’s. Monty cooked a huge beef tenderloin, and we talked, sang songs and drank. I managed to get back to my hotel about 1am the next morning.

Usually when I travel home from Europe my flight will leave around noon and I get back in the early afternoon local time. For some reason those flights were over $1000 more than the Finnair flight that left at 5pm, so I returned to Monty’s on Friday morning to visit for a few hours.

I loved the fact that Monty was so welcoming and also that he and his family keep a lot of animals (we do the same). In addition to six cats and three dogs, there is a boa constrictor named Monty Python who is about two meters long. The story I heard was that it was a gift given to a family member that ended up at Monty’s. They originally thought it was a python but later learned it was a boa, but the name stuck.

The trip home was uneventful except for the fact that I got home home close to 2am and I ended up catching a bad case of influenza. To my knowledge no one else at the conference got sick, for which I’m happy, and while it knocked me out of commission for almost two weeks it was worth it.

The last day of SCaLE was bittersweet, as I didn’t want it to be over but I was also ready to head home.

After stopping by the booth I was eager to visit a session on OpenNMS presented by my friend Jeff Gehlbach.

Jeff has stepped up in the the presenter role I used to have, and he did a very good job of covering what network flows are, the different types and why they are important.

Back in the Exhibit Hall I was happy to learn that the AWS booth had won the “Most Memorable” award.

Hats off to Spot and Ashley for coming up with such a cool concept and creating a great space for people to hang out.

At 1:30pm we held a raffle for a pretty nice 3D printer. You had to be present to win and there was a lot of interest.

Then it was time to tear down the booth as the Exhibit Hall closed at 2pm.

This gave us time to get to the closing keynote by Internet pioneer Vint Cerf.

For someone who recently turned 79 he was a dynamic and entertaining speaker, and it was fun to listen to his stories on creating ARPANET, and how it grew into the public Internet we use today.

He also mentioned Jon Postel several times. I had an e-mail correspondence with Jon in the mid-1990s when I was trying to wrap my brain around the process for getting an “enterprise number” from IANA. I didn’t realize until after his untimely death who he was, and I’m still impressed at how much time he was willing to give a newbie like me.

While I enjoyed the presentation, I did regret that we ran out of time for details on his last slide, which concerned “unfinished business”.

I mean, I get it. Each of the six topics on the slide could be a talk on its own, but I was very curious to hear his thoughts on fixing things such as disinformation. I love living in a world with almost instant access to information and the ability to connect with others, but there are problems, too, and I’m not sure we have the solutions.

All in all I am extremely happy to have been able to attend SCaLE. I’m still not comfortable in crowds and I was a little put out that not everyone in attendance decided to honor the mask policy. I talked with the SCaLE staff and they told me they were doing all they could, but even when people were reminded to mask up they tended to remove them as soon as the staff member walked away.

I was especially unhappy when I saw sponsors going maskless. On the one hand I am happy for their support of SCaLE, but on the other when you are standing in front of your company logo showing a disregard for the safety of your potential customers, it sends a bad message.

I’m not bringing this up to start a debate on the efficacy of masks, as I realize that they provide varying degrees of protection depending on type and use, but if your staff isn’t willing to abide by the conference rules, perhaps you just shouldn’t be there.

Note that I did refrain from posting the pictures I took of specific sponsors since it really wouldn’t change anything. I must be getting soft in my old age.

In any case I hope this is a non-issue for SCaLE 20x in Pasadena next March. I’m not optimistic that the pandemic will be over but for me the risk was worth the benefit, and I can’t wait to return.

Day Three of SCaLE kicked off the start of the main conference, which meant I spent most of the day in the AWS booth.

Traffic was pretty good and I got to talk with a lot of interesting people. I did take a break around 2pm and noticed from Twitter that I was missing a talk by Frank Karlitschek of Nextcloud fame, so I skedaddled over to his room to catch it.

It was pretty good. It focused on how copyleft-style licenses are often better for business since they level the playing field for all contributors, versus a number of newer licenses that are more “source available” instead of “open source”.

Please note that I’m an unabashed Nextcloud fanboy so I have some biases. (grin)

The big evening event was “Game Night” where they turned the basement ballrooms into a big gaming playground. From the classics such as checkers and chess, to Vegas-style games such as roulette and blackjack, up to the most modern of games using VR, there was something for everyone.

AWS sponsored the music for the event, and I was eager to see MC Frontalot perform. He didn’t disappoint.

He did an hour-long set spanning the classics to the newer stuff, including “Secrets From the Future” featuring a video generated using AI.

Afterward he hung out at the merch table to chat with folks, and I got to spend some time with a new friend named Silona Bonewald.

I was introduced to Silona through Spot as she was on the same hotel shuttle bus when we arrived on Wednesday evening. She is in charge of open source at IEEE as well as being a Burner, and I always look forward to chance to talk with her.

Today is the final day of the conference, and remember if you are reading this before 1:30pm PDT there is a raffle for an awesome 3D printer at the AWS booth, so come by to get your ticket.

This is the first conference since joining AWS that I have booth duty, so I won’t be able to spend as much time in the sessions as I would like, but I did want to catch one of the first sessions of the day which was “Speedrunning Kubernetes”.

The main reason I wanted to see this talk was to see Kat Cosgrove in action. Prior to coming to AWS I didn’t know about her but I ended up following her on Twitter and found that she has strong opinions, and I tend to like people who have strong opinions. I figured the presentation would be entertaining and that I might learn something.

I wasn’t disappointed.

The title alludes to a “speedrun” which is an attempt to complete a video game as quickly as possible. The goal of this talk was to bring up a working Kubernetes cluster as if you were doing a speedrun. It also included one of the more … unusual … analogies I’ve seen in a technical presentation (including my own) by using a Chihuahua as a metaphor.

If the goal is to provide the “cheeseburger” application, consisting of the bun service, the patty service, the cheese service, the mustard service, etc., each instance of the application (i.e. each burger) can be considered a “pod”. There are two pods under each foot of the dog representing two-pod “nodes” and the dog forms the control plane.

Remember, now that you’ve seen it, you can’t unsee it.

That was the only session I made on Day Two, but I did get some time to wander around the Exhibit Hall. The Software Freedom Conservancy had a booth, and since they are one of my favorite organizations I stopped by to chat with Pono Takamori. I know a number of folks that work there and they serve as almost a reference implementation for trying to live using 100% free software. Pono was telling me that it was getting almost impossible to find a totally free mobile wireless solution since 3G went away, as all of the modern modems tend to use binary blobs.

Now, when these exhibit halls are being set up, the “booths” are laid out with little generic signs showing the owner of the booth, and most of the time they eventually get covered up once the booth is complete.

I know the Sun acquisition was a long time ago, but I still get cognitive dissonance when I see a MySQL sign next to an Oracle one.

The AWS booth for this conference is really awesome. I bow down to the genius that is Spot Callaway, and he pitched a booth design that was to invoke a teenage geek’s basement, where one might play video games and Dungeons and Dragons (think Stranger Things). The walls of the booth are made to look like brick, and there are chairs, a couch and an SNES console emulator.

The featured AWS project for this conference is Bottlerocket, and I got to learn a bit about it and meet members of the team. Bottlerocket is a minimal operating system designed just to run containers. I compared it to LibreELEC, which is a purpose-built O/S that I use to run Kodi, and while it was explained to me that I was oversimplifying things a bit, it was otherwise a good analogy.

While it is, of course, being used withing AWS, it is a 100% open source project and you can get the code on Github, and the hope is that others will find it valuable and will get involved with the community. If this is something you’re into, stop by the booth and say “hi”.

Speaking of stopping by the booth, we do have some tasty sodas and Bottlerocket branded bottle openers, but the big giveaway is an awesome 3D printer. Get a raffle ticket and stop by the booth at 1:30pm on Sunday for the drawing (you must be present to win).

AWS employees are not eligible to participate. (sniff)

I am back at the Southern California Linux Expo (SCaLE) for the first time in many years, and I was surprised at how happy this makes me. It is always a well run conference and it tends to bring a lot of people I like together in one place, which means I get to meet a lot more people to like as well.

The main SCaLE sessions occur over the weekend, but there are a lot of cool things that happen in the days before. For Thursday, AWS sponsored Cloud Native Builder Day to showcase some of the amazing open source technologies one can use to solve a number of challenges, and I was eager to learn about them.

But before that I needed to get registered. The first step was to show proof of vaccination. While I am thankful that we can have these events, COVID is still a thing and the organizers are doing all they can to mitigate the risk to the conference attendees. Since I’m an old I’ve had two shots and two boosters but the darn thing keeps mutating.

Once past that I headed upstairs where I could use the self check-in kiosks. It was pretty simple to sign in and get my badge printed, and then it was just a short trip down the hall to pick up the conference “swag bag” which included the badge holder and lanyard.

The only change I would make to the process is that once you printed your badge, you should really hit the “close window” button on the screen, as there is a “back” button that could allow the next person who registers to see your name and e-mail. No biggie, but the security nerd in me always thinks about these things.

The conference spans two floors. The Exhibit Hall with the sponsor booths is on the ground floor behind registration (it is technically in the Plaza Ballroom so I just followed the signs for “ballrooms”) while the sessions are on the second floor along with registration. AWS is going to have a pretty cool booth this year.

As an AWS employee I guess I should say that we always have a cool booth (grin) but I especially like the idea behind this one, despite the fact that we were unable to get a mounted deer head (seriously). It’s booth numbers 300, 302 and 304 if you want to swing by, and for those of you who couldn’t make it I’ll be sure to post about it later.

Cloud Native Builder Day showcased three different open source projects, the first one being Triggermesh. This was presented by Jeff Naef who I immediately liked as he was the first to notice that my mask is made by K&N, a company known for their high-end automotive airflow products. He loves performance automobiles as well as open source (he was wearing a Snap-On tools hat) so I knew we would get along.

In dealing with cloud native technologies, a lot of the workflow is event driven. Triggermesh lets you seamlessly link together sources and targets for events, normalizing and enriching them along the way. While it does support the ability to create functions using code (in a variety of languages) a lot of the implementation can be done just through configuration.

In one example the data was encoded in base64, and a person asked if Triggermesh could render that in clear text. Jeff was like, sure, and he bravely set out to implement that as we watched. He got really close, but in any case deserves kudos for the attempt, especially considering he was holding a microphone with one hand the entire time.

The next speaker was Zoe Steinkamp from InfluxDB. I first met Zoe at the Open Source Summit in Austin and she is one of my favorite new acquaintances I’ve met through my job at AWS.

Now full disclosure: I missed the first half of her presentation.

SCaLE has done something delightful with the schedule, which is allowing 30 minutes between talks. I’ve talked about this before but this lets speakers switch out without the usual urgency, allows more time for attendees to interact with the speaker after the talk, and improves the hallway track.

I thought I had enough time to grab lunch, which was In-N-Out that Spot had brought for me. We don’t have In-N-Out in North Carolina so I rarely pass up a chance to get it, and I figured I could be back in time. I was wrong. But I did slip into the back of the room which is why this picture isn’t as close as the others.

I used to work on an open source project that relied heavily on time series data, so I’m a bit of a time series data geek. Every time I see a presentation on InfluxDB I learn more things to like about it. This time I found out that it is possible to get started with it without being a programmer. A lot of people in the data science field aren’t coders, but they can send their data to InfluxDB pretty easily. The folks at Influx have created InfluxDB University as a free resource to get the most out of their solution, and while I haven’t gone through it yet it looks really comprehensive.

The final presenter was Matt Overstreet from Datastax. Datastax focuses on providing solutions around the Apache Cassandra project, which is a distributed “NoSQL” database.

When most people hear the word “database” they think of relational databases. This is a data structure usually based on “rows” of data made up of “fields” and indexed by a primary key. One then uses something like the Structured Query Language (SQL) to retrieve values from those fields. This is all well and good but it tends to be extremely monolithic, which doesn’t work well in today’s distributed cloud environment.

Think about it. In a datacenter you might have sub-millisecond latency, so a query can be returned quickly. Move that datacenter across the country, and now it your latency is, say, 100ms. Move that to the other side of the world and, well, you get the picture. Now if you only have a few queries that might be okay, but when you consider thousands and then millions of queries, the response time of your application is going to take a hit.

Cassandra allows you to distribute that data both within a datacenter (for reliability) and also regionally. You can then put your data near your customers, improving their experience.

I was already sold on Cassandra (we used it at OpenNMS) but what I learned from this presentation was the wonderfulness that is “k8ssandra” (kate-sandra). This is Cassandra but running in Kubernetes. If you have ever had to extend and expand a Cassandra cluster, you know that while it isn’t super difficult there are a number of gotchas that can cause problems. What if you could automate it? Matt showed us an example that let him spin up (and tear down) an 800 node cluster in minutes.

Cool, huh?

The first day of SCaLE 19x was a blast, and I am eager to see what the rest of the week brings.

I always feel a little sad on the last day of any conference, and Open Source Summit was no different. It seems like the week went by too fast.

With the Sponsor Showcase closing on Thursday, attendance at the Friday keynotes was light, but those of us that showed up got to hear some pretty cool presentations.

The first one was from Rachel Rose, who supervises R&D at Industrial Light and Magic. As a fanboy of ILM I was very eager to hear what she had to say, and she didn’t disappoint. (sorry about the unflattering picture but I took three and they were all bad)

In the past a lot of special effects that combine computer generated imagery (CGI) and live action are created separately. The live action actors perform in front of a green screen and the CGI backgrounds are added later. Technology has advanced to the point that the cutting edge now involves live action sets that are surrounded by an enormous, curved LED screens, and the backgrounds are projected as the actors perform.

This presents a number of challenges as the backgrounds may need to change as the camera moves, but it provides a much better experience for the actors and the audience.

The tie-in to open source is that a lot of the libraries used the creation of these effects are now open. In fact, the Academy of Motion Picture Arts and Sciences (the people responsible for the Oscars) along with the Linux Foundation have sponsored the Academy Software Foundation (ASWF) to act as a steward for the “content creation industry’s open source software base”. The projects under the ASWF fall into one of two tiers: Adopted and Incubation. Currently there are four projects that are mature enough to be adopted and several more in the incubation stage.

A lot of this was so specific to the industry that it went over my head, but I could understand the OpenEXR project, which provides a reference implementation of the EXR file format for storing high quality images.

She then went on to talk about Stagecraft, which is the name of the ILM platform for producing content. I would love to be able to visit one day. It would be so cool to see a feature being made with the CGI, sets and actors all integrated.

The next speaker was Vini Jaiswal, Developer Advocate for Databricks. I had seen a cool Databricks presentation back on Day 2 and the first part was similar, but Jaiswal skipped the in-depth technical details and focused more on features and adoption. A rather large number of companies are using the Delta Lake technology as a way to apply business intelligence to data lakes, and as the need to analyze normally unstructured data becomes more important, I expect to see even more organizations adopt it.

The third presentation was a video by Dmitry Vinnik of Meta on measuring open source project health.

Begin rant.

To be honest I was a little unhappy to see a video as a keynote. It was the only one for the entire week and I have to admit I kind of tuned it out. It wasn’t even novel, as he has given it at least twice before. The video we were shown is available on Youtube from a conference earlier in the month and he posted another one dated June 24th from the Python Web Conference (while it has a different splash screen it looks to be the same presentation).

Look, I’ve given the same talk multiple times at different conferences, so I get it. But to me keynotes are special and should be unique. I was insulted that I bothered to show up in person, wear a mask, get my temperature checked each day, and I expected something better than a video I could have watched at home.

Note: Rachel Rose played a video as part of her presentation and that’s totally cool, as she didn’t “phone in” the rest of it.

Okay, end rant.

The next two presenters were very inspiring young people, and it was nice to have them included as part of the program.

The first speaker was Alena Analeigh, an amazing young woman who, among other achievements, has been accepted to medical school at age 13 (note that in trying to find a reference for that I came up blank, except for her twitter bio, so if you have one please let me know and I can update this post).

Med school is just one of her achievements. She also founded The Brown STEM Girls as an organization to get more women of color interested in science, technology, engineering and math. She stated that while men make up 52% of the workforce, they represent 76% of people employed in STEM fields.

My love of such things was fostered at an early age, and programs like hers are a great step to encourage young women of color to get interested in and eventually pursue careers in STEM.

While she seemed a little nervous and tentative while presenting, the final speaker of the morning was the exact opposite. At 11 years old, I could listen to Orion Jean speak for hours.

Orion also has a number of accolades, including Time Magazine’s “Kid of the Year“. He got his start as the winner of a speech contest sponsored by Think Kindness, and since then has started the Race to Kindness (“a race where everybody wins”) to spread kindness around the world.

To help inspire acts of kindness he uses the acronym K.I.N.D.:

• Keep Your Eyes Open: Look for opportunities to be kind to others. One example he used is one I actually practice. If you are in line to check out at the store, and you see a person with a lot less items than you, while not offer to let them check out first?
• Include Others: No one can effect change alone. Get others involved.
• Nothing Is Too Small: One thing that keeps us from spreading kindness is that we can try to think too big. Even small acts of kindness can have a huge impact.
• Do Something About It: Take action. Nothing can change if we do nothing.

After the keynotes I had to focus on some work stuff that I had let languish for the week, so I didn’t make it to any of the presentations, but overall I was happy with my first conference in three years.

There were a few people that attended who tested positive for COVID, so I plan to take some precautions when I get home and hope that the steps the Linux Foundation took to mitigate infection worked. So far I’ve tested negative twice, and I’ll probably take another test on Monday.

My next conference will be SCaLE in Los Angeles at the end of July, and I plan to be in Dublin, Ireland for Open Source Summit – Europe. If you are comfortable getting out and about I hope to see you there.