2016 Dev-Jam: Day 3

It’s hard to believe this year’s Dev-Jam is half over. After months of planning it seems to go by so fast.

One of the goals I had this week was to understand more about the OpenNMS Documentation Project. For years I’ve been saying that OpenNMS documentation sucks like most open source projects, but I can’t say that any more. It has actually become quite mature. There is a detailed installation guide, a users guide, and administrators guide and a guide for developers. Each release the docs are compiled right alongside the code, and it even rates its own section on the new website.

Web Site Docs Page

It’s written in AsciiDoc, and all of the documentation is version controlled and kept in git.

Ronny Trommer is one of the leads on the documentation project, and I asked him to spend some time with me to explain how everything is organized.

Ronny Trommer

Of the four main guides, the installation guide is almost complete. Everything else is constantly improving, with the user guide aimed at people working through the GUI and the administration guide is more focused on configuration. For example, the discussion of the path outage feature is in the users guide but how to turn it on is in the admin guide.

There is even something for everyone in the developers guide (I am the first to state I am not a developer). One section details the style rules for documentation, in great detail. For example, in order to manage changes, each sentence should be on a single line. That way a small change to, say, a misspelled word, doesn’t cause a huge diff. Also, we are limited as to the types of images we can display, so people are encouraged to upload the raw “source” image as well as an exported one to save time in the future should someone want to edit it.

It is really well done and now I’m eager to start contributing.

Speaking of well done, Jonathan has figured out what is keeping OpenNMS from using the latest version of OTRS (and he’s sent a patch over to them) and Jesse showed me some amazing work he’s done on the Minion code.

We’ve been struggling to figure out how to implement the Minion code since we want to be able to run it on tiny machines like the Raspberry Pi, but since OpenNMS is written in Java there is a lot of overhead to using that language on these smaller systems. He re-wrote it in Go and then uploaded it to a device on my home network. At only 5.6MB it’s tiny, and yet it was able to do discovery as well as data collection (including NRTG). Sheer awesomeness.

Wednesday was also Twins night.

Twins Tickets

For several years now we’ve been going as a group to see the Minnesota Twins baseball team play at Target Field. It’s a lot of fun, although this year the Germans decided that they’d had enough of baseball and spent the time wandering around downtown Minneapolis.

At first I thought they had the right idea, as the Braves went up 4 to 0 in the first and by the top of the fourth were leading 7 to 0. However, the Twins rallied and made it interesting, although they did end up losing 9 to 7.

Our seats were out in left field, ‘natch.

Twins Tickets

2016 Dev-Jam: Day 2

By Day Two people have settled into a rhythm. Get up, eat breakfast, start hacking on OpenNMS. I tend to start my day with these blog posts.

It’s nice to have most of the team together. Remember, OpenNMS is over 15 years old so there is a lot of different technology in the monitoring platform. I think David counted 18 different libraries and tools in the GUI alone, so there was a meeting held to discuss cleaning that up and settling on a much smaller set moving forward.

In any case ReST will play a huge role. OpenNMS Compass is built entirely on ReST, and so the next generation GUI will do the same. It makes integrating with OpenNMS simple, as Antonio demonstrated in a provisioning dashboard he wrote for one of his customers in Italy.

Antonio Teaching

They needed an easier way to manage their ten thousand plus devices, so he was able to use the ReST interface to build out exactly what they wanted. And of course the source is open.

Several years ago we started a tradition of having a local restaurant, Brasa, cater dinner one night. This year it was Tuesday, and it is always the best meal of the week.

Antonio Teaching

As we were getting ready to eat, Alex Hoogerhuis, a big supporter of OpenNMS who lives in Norway, decided to join us via our Double Robotics robot, Ulfbot. It worked flawlessly, and he was the best first time driver we’ve had. Ben, Jeff and Jonathan joined him for a picture.

Alex and Team

We like using the Yudof Hall Club Room for Dev-Jam for a number of reasons, one includes the big patio overlooking the river with picnic tables. Alex was able to drive around and spend some time with the rest of the team, although we had to lift him up to see over the wall to the Mississippi (we also had to carry him in when the wind picked up – heh).

Alex at Dinner

After dinner people kept working (DJ was up until nearly 2am chasing a bug) but we also took a break to watch Deadpool. It’s why “Dev-Jam” rhymes with “fun”.

Review: X-Arcade Gaming Cabinet

Last year I wanted to do something special for the team to commemorate the release of OpenNMS Meridian.

Since all the cool kids in Silicon Valley have access to a classic arcade machine, I decided to buy one for the office. I did a lot of research and I settled on one from X-Arcade.

X-Arcade Machine

The main reasons were that it looked well-made but it also included all of my favorites, including Pac-Man, Galaga and Tempest.

X-Arcade Games

The final piece that sold me on it was the ability to add your own graphics. I went to Jessica, our Graphic Designer, and she put together this wonderful graphic done in the classic eight-bit “platformer” style and featuring all the employees.

X-Arcade Graphic

Ulf took the role of Donkey Kong, and here is the picture meant to represent me:

X-Arcade Tarus

The “Tank Stick” controls are solid and responsive, although I did end up adding a spinner since none of the controls really worked for Tempest.

When you order one of these things, they stress that you need to make sure it arrives safely. Seriously, like four times, in big bold letters, they state you should check the machine on delivery.

I was going to be out of town when it arrived, so I made sure to tell the person checking in the delivery to make sure it was okay (i.e. take it out of the box).

They didn’t (the box looked “fine”) and so we ended up with this:

X-Arcade Cracked Top

(sigh)

Outside of that, everything arrived in working order. You get a small Dell desktop running Windows with the software pre-installed, but you also get CDs with all the games that are included with the system. It’s a little bit of a pain to set up since the instructions are a little vague, but after about an hour or so I had it up and running.

Anyway, it is real fun to play. It supports MAME games, Sega games, Atari 2600 games and even that short lived laserdisc franchise “Dragon’s Lair”. You can copy other games to the system if you have them, although scrolling through the menu can get a bit tiring if you have a long list of titles.

We had an issue with the CRT about 11 months after buying the system. I came back from a business trip to find the thing dark (it never goes dark, if the computer is hung for some reason you’ll still see a “no signal” graphic on the monitor”). Turns out the CRT had died, but they sent us a replacement under warranty and hassle free. It took about an hour to replace (those instructions were pretty detailed) and it worked better than ever afterward.

This motivated me to consider fixing the top. When we had the system apart to replace the monitor, I noticed that the top was a) the only thing broken and b) held on with eight screws. I contacted them about a replacement piece and to my surprise it arrived two days later – no charge.

The only issue I have remaining with the system is the fact that it is Windows-based. This seems to be the perfect application for a small solid-state Linux box, but I haven’t had the time to investigate a migration. Instead I just turned off or removed as much software as I could (all the Dell Update stuff kept popping up in the middle of playing a game) and so far so good.

I am very happy with the product and extremely happy with the company behind it. If you are in the market for such a cabinet, please check them out.

2016 Dev-Jam: Day 1

Dev-Jam officially started on Monday at 10am, where I did my usual kick-off speech before turning it over to Seth and Jesse who handle the technical side of things.

Yesterday I stated that this was our tenth Dev-Jam at UMN. I forgot that the first one was held at my house, so this actually makes this the ninth (we’ve still had eleven since 2005).

Yudof Club Room

Everyone went around the room and talked about the things they wanted to work on this week. A lot of them focused on Minion, a technology rather unique to OpenNMS. A Minion is a a Karaf container that implements features for remote monitoring. It is key for OpenNMS to be able to scale to the Internet of Things (IoT) level of millions of devices and billions of metrics. And speaking of IoT, Ken turned me on to openHAB which is something I need to check out.

Yudof Kitchen

It is often hard for me to describe Dev-Jam to other people, as it is truly a lightly structured “un-conference”. In a great example of the Open Source Way it is very self-organizing, and I look forward to Friday when everyone presents what they have done.

Some of the Germans

We did have Alex Finger, one of the creators of the OpenNMS Foundation, join use via robot. He was having some sound issues and I think he did get stymied by the robot’s lack of hands when he came across a door, but it was cool he was able to visit from Europe.

Alex on the Robot

We use this week for planning and sharing, so Jesse took some time to go over the Business Service Monitor (BSM) which allows you to create a “business level” view of your services versus just the devices themselves. It is fully implemented via ReST and it pretty powerful, although as with a lot of things OpenNMS that very power can add complexity. I’m hoping our community will find great uses for it.

jesse and BSM

That evening about half of us walked to a theatre to see Star Trek Beyond. Most of us disliked it and I posted a negative review, but it was fun to go out with my friends.

2016 Dev-Jam: Day 0

♬ It’s the most wonderful time of the year ♬

Ah yes, it’s Dev-Jam time, where we descend onto the campus of the University of Minnesota, Twin Cities, for a week of OpenNMS goodness.

This is our eleventh annual Dev-Jam and our tenth at UMN. They are really good hosts so we’ve found it hard to look elsewhere for a place to hold the conference.

This is not a user’s conference. That is coming up in September. Instead, this is a chance for the core contributors of OpenNMS, and those people who’d like to become core contributors, to get together, share and determine the direction of OpenNMS for another year.

This year we are just shy of 30 people from four different countries: The US, the UK, Italy and India. Alejandro and his wife Carolina are now permanent residents of the US so I can’t really count them as being from Venezuela any more, and that happened directly through his involvement with Dev Jam. We’ve had more people but 30 seems to be the magic number (one year we had 40 and it was much harder to manage)

MSP sign at airport

My trip to MSP was uneventful. I flew through Dallas even though there is a direct RDU->MSP flight on Delta since I’m extremely close to Lifetime Platinum status on American Airlines. Also, AA has added a cool feature on their mobile app that lets me track my bags. This was important since I was shipping a box of four 12-packs of Cheerwine – a Dev-Jam favorite and as always a target for TSA inspection (apparently a 40+ pound box of soda is suspicious). Everything got here fine.

Including Ulf:

Ulf in Admiral's Club

Ulf is the OpenNMS mascot and he, too, is a product of Dev-Jam. Many years ago Craig Miskell came to Dev-Jam from New Zealand. He brought this plush toy and gave it to the Germans, who named him “Ulf”. Since then he has been around the world spreading the Good News about OpenNMS, so it wouldn’t be Dev-Jam without him.

We stay in a dorm called Yudof Hall where we take over the Club Room, a large room on the ground floor that includes a kitchen and an area with sofas and a television. In the middle we set up tables where we work, and due to UMN being a top-tier university we have great bandwidth. There is a huge brick patio next to it that looks out over the Mississippi River. It’s a very nice place to spend the week.

Speaking of the Mississippi, we crossed it last night to our usual kick-off spot, the Town Hall Brewery. As a cocktail aficionado, I was happy to see some craft cocktails on the menu, and a number of us tried the “Hallbach”, their take on the Seelbach Cocktail:

Hallbach Cocktail

It was very nice, as they used a high proof bourbon and replaced the champagne with sparkling cider.

We like Town Hall since we can seat 30 people. We do cater in as well as go out. The new light rail service to campus makes getting around easy, especially to the Mall of America and Target Field.

Speaking of baseball, we’re all going to the game on Wednesday. If you are in the area and want to join us, I should have a couple of tickets available. Just drop me a note. We also brought along the Ulfbot, which is a tele-presence robot so do the note dropping thing if you want to “visit”.

Dev-Jam!

New Fancy Website for www.opennms.org

As some of you may have noticed, a little while ago the OpenNMS Project website got updated to a new, fancy, responsive version.

OpenNMS Platform

This was mainly the work of Ronny Trommer with a big assist from our graphic designer, Jessica.

We are often so busy working on the code we often forget how important it is to tell people about what we are doing. Most people who take the time to learn about the project realize how awesome it is, but it can be hard to get over that first hump in the learning curve.

I hope that the new site will both reflect the benefits of using OpenNMS as well as the work of the community behind it.

OpenNMS Meridian 2016 Released

I am woefully behind on blog posts, so please forgive the latency in posting about Meridian 2016.

As you know, early last year we split OpenNMS into two flavors: Horizon and Meridian. The goal was to create a faster release cycle for OpenNMS while still providing a stable and supportable version for those who didn’t need the latest features.

This has worked out extremely well. While there used to be eighteen months or so between major releases, we did five versions of Horizon in the same amount of time. That has led to the rapid development of such features as the Newts integration and the Business Service Monitor (BSM).

But that doesn’t mean the features in Horizon are perfect on Day One. For example, one early adopter of the Newts integration in Horizon 17 helped us find a couple of major performance issues that were corrected by the time Meridian 2016 came out.

The Meridian line is supported for three years. So, if you are using Meridian 2015 and don’t need any of the features in Meridian 2016, you don’t need to upgrade. Major performance issues, all security issues and most of the new configurations will be backported to that release until Meridian 2018 comes out.

Compare and contrast that with Horizon: once Horizon 18 was released all work stopped on Horizon 17. This means a much more rapid upgrade cycle. The upside being that Horizon users get to see all the new shiny features first.

Meridian 2016 is based on Horizon 17, which has been out since the beginning of the year and has been highly vetted. Users of Horizon 17 or earlier should have an easy migration path.

I’m very happy that the team has consistently delivered on both Horizon and Meridian releases. It is hoped that this new model will both keep OpenNMS on the cutting edge of the network monitoring space while providing a more stable option for those with environments that require it.

Upgrading Linux Mint 17.3 to Mint 18 In Place

Okay, I thought I could wait, but I couldn’t, so yesterday I decided to do an “in place” upgrade of my office desktop from Linux Mint 17.3 to Mint 18.

It didn’t go smoothly.

First, let me stress that the Linux Mint community strongly recommends a fresh install every time you upgrade from one release to another, and especially when it is from one major release, like Mint 17, to another, i.e. Mint 18. They ask you to backup your home directory and package lists, base the system and then restore. The problem is that I often make a lot of changes to my system which usually involves editing files in the system /etc directory, and this doesn’t capture that.

One thing I’ve always loved about Debian is the ability to upgrade in place (and often remotely) and this holds true for Debian-based distros like Ubuntu and Mint. So I was determined to try it out.

I found a couple of posts that suggested all you need to do is replace “rosa” with “sarah” in your repository file, and then do a “apt-get update” followed by an “apt-get dist-upgrade”. That doesn’t work, as I found out, because Mint 18 is based on Xenial (Ubuntu 16.04) and not Trusty (Ubuntu 14.04). Thus, you also need to replace every instance of “trusty” with “xenial” to get it to work.

Finally, once I got that working, I couldn’t get into the graphical desktop. Cinnamon wouldn’t load. It turns out Cinnamon is in a “backport” branch for some reason, so I had to add that to my repository file as well.

To save trouble for anyone else wanting to do this, here is my current /etc/apt/sources.list.d/official-package-repositories.list file:

deb http://packages.linuxmint.com sarah main upstream import backport #id:linuxmint_main
# deb http://extra.linuxmint.com sarah main #id:linuxmint_extra

deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu/ xenial-security main restricted universe multiverse
deb http://archive.canonical.com/ubuntu/ xenial partner

Note that I commented out the “extra” repository since one doesn’t exist for sarah yet.

The upgrade took a long time. We have a decent connection to the Internet at the office and still it was over an hour to download the packages. There were a number of conflicts I had to deal with, but overall that part of the process was smooth.

Things seem to be working, and the system seems a little faster but that could just be me wanting it to be faster. Once again many thanks to the Mint team for making this possible.

MC Frontalot and The Doubleclicks at All Things Open

I am happy to finally be able to confirm that MC Frontalot and his band, along with The Doubleclicks, will be playing an exclusive show during the All Things Open conference in October. The OpenNMS Group, at great expense (seriously, this is like our entire marketing budget for the year), has secured these two great acts to help celebrate all things open, and All Things Open.

MC Frontalot

I first met Damian (aka Frontalot) back in 2012 when I hired him to play at the Ohio Linuxfest. I subscribe to the Chris Dibona theory that open source business should give back to the community (he once described his job as “giving money to his friends”) and thus I thought it would be cool to introduce the übernerd Frontalot to the open source world.

We hit it off and now we’ve hired him a number of times. The last time was for OSCON in 2015, where we decided to bring in the entire band. What an eye-opening experience that was. A lot of tech firms talk about “synergy” – the situation when the whole is greater than the sum of its parts – but Front with his band takes the Frontalot experience to a whole new level.

Also at the OSCON show we were able to get The Doubleclicks to open. This duo of sisters, Angela and Aubrey Webber, bring a quirky sensibility to geek culture and were the perfect opening act.

Now, I love open source conferences, but I overdid it last year. So this year I’m on a hiatus and have been to *zero* shows, but I made an exception for All Things Open. First, it’s in my home city of Raleigh, North Carolina, which is also home to Red Hat. We like to think of the area as the hot bed of open source if not its heart. Second, the conference is organized by Todd Lewis, the Nicest Man in Open Source™. He spends his life making the world a better place and it is reflected in his show. We couldn’t think of a better way to celebrate that then to bring in some top entertainment for the attendees.

That’s right: there are only two ways to get in to this show. The easiest is to register for the conference, as the conference badge is what you’ll need to get in to the venue. The second way is to ask us nicely, but we’ll probably ask you to prove your dedication to free and open source software by performing a task along the lines of a Labor of Hercules, except ours will most likely be obscenely biological.

Seriously, if you care about FOSS you don’t want to miss All Things Open, so register.

If you are unfamiliar with the work of MC Frontalot, may I suggest you check out “Stoop Sale” and “Critical Hit“, or if you’re Old Skool like me, watch “It Is Pitch Dark“. His most recent album was about fairy tales (think of it as antique superhero origin stories). Check out “Start Over” or better yet the version of “Shudders” featuring the OpenNMS mascot, Ulf.

As for The Doubleclicks, you can browse most of their catalog on their website. One song that really resonates with me, especially at conferences, is “Nothing to Prove” which I hope they’ll do at the show.

Oh, and I saved the best for last, Front has been working on a free software song. Yup, he is bringing his mastery of rhymes to bear on the conflict between “free as in beer” and “free as in liberty” and its world premiere will be, you guessed it, at All Things Open.

The show will be held at King’s Barcade, just a couple of blocks from the conference, on Wednesday night the 26th of October. You don’t want to miss it.

First Thoughts on Linux Mint 18 “Sarah”

I am a big fan of Linux Mint and I look forward to every release. This week Mint 18 “Sarah” was released. I decided to try it out on my Dell XPS 13 laptop since it is the easiest machine of mine to base and they really haven’t suggested an upgrade path. The one article I was able to find suggested a clean install, which is what I did.

First, I backed up my home directory, which is where most of my stuff lives, and I backed up the system /etc directory since I’m always making a change there and forgetting that I need it (usually concerning setting up the network interface as a bridge).

I then installed a fresh copy of Mint 18. Now they brag that the HiDPI support has improved (as I will grouse later, so does everyone else) but it hasn’t. So the first thing I did was to go to Preferences -> General and set “User interface scaling” to “Double”. This worked pretty well in Mint 17 and it seems to be fine in Mint 18 too.

I then did a basic install (I used a USB dongle to connect to a wired network since I didn’t want to mess with the Broadcom drivers at this point) and chose to encrypt the entire hard drive, which is something I usually do on laptops.

I hit my first snag when I rebooted. The boot cycle would hang at the password screen to decrypt the drive. In Mint 17 the password prompt would be on top of the “LM” logo. I would type in the password and it would boot. Now the “LM” logo has five little dots under it, like the Ubuntu boot screen, and the password prompt is below that. It’s just that it won’t accept input. If I boot in recovery mode, the password prompt is from the command line and works fine.

(sigh)

This seems to be a problem introduced with Ubuntu 16.04. Well, before I dropped back down to Mint 17 I decided to try out that distro as well as Kubuntu. My laptop was based in any case.

I ran into the usual HiDPI problems with both of those. I really, really want to like Kubuntu but with my dense screen I can’t make out anything and thus I can’t find the option to scale it. Ubuntu’s Unity was easier as it has a little sliding scaler, but when I got it to a resolution I liked many of the icon labels were clipped, just like last time I looked at it.

(sigh)

Then it dawned on my that I could just install Mint 18 but see if encrypting just my home directory would work. It did, so for now I’m using Mint 18 without full disk encryption. The next step was to install the proprietary Broadcom driver and then wireless worked.

Next, I edited /etc/fstab and added my backup NFS mount entry, mounted the drive and started restoring my home directory. That went smoothly, until I decided to reboot.

The laptop just hung at the boot screen.

Now there is a bug in Dell BIOS that if I try to boot with a USB network adapter plugged in, it erases the EFI entry for “ubuntu” and I have to go into setup and manually re-add it. Thus I was disconnecting the dongle for every reboot. On a whim I plugged it back in and the system booted. This led me to believe that there was an issue with the NFS mount in /etc/fstab, and that’s what the problem turned out to be.

The problem is that systemd likes to get its little hands into everything, so it tries to mount the volume before the wireless network is initialized. The solution is to add a special option that will cause systemd to automount the volume when it is first requested. Here is what worked:

172.20.10.5:/volume1/Backups /media/backups nfs noauto,x-systemd.automount,nouser,rsize=8192,wsize=8192,atime,rw,dev,exec,suid 0

The key bits are “noauto,x-systemd.automount”.

With that out of the way, I added mounts for my music and my video collection. That’s when I noticed a new weirdness in Cinnamon: dual icons on the desktop. I have set the desktop option to display icons for mounted file systems and now I get two of them for each remote mount point.

Double Desktop Icons

Annoying and I haven’t found a solution, so I just turned that option back off.

Now I was ready to play with the laptop. I’m often criticized for buying brand new hardware and expecting solid Linux support (yeah, you, Eric) but this laptop has been out for over a year. Still the trackpad is a little wonky – the cursor tends to jump to the lower right hand corner. Mint 18 ships with a 4.4 kernel but I had been using Mint 17 with a 4.6 kernel. One of the features of 4.6 is “Dell laptop improvements” so while I was hoping 4.4 would work for me (and that the features I needed would have been backported) it isn’t so. I installed 4.6 and my trackpad problems went away.

The final issue I needed to fix concerned ssh. I use ssh-agent and keys to access a lot of my remote servers, and it wasn’t working on Mint 18. Usually this is a permissions issue, but I compared the laptop to a working configuration on my desktop and the permissions were identical.

The error I got was:

debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0

It turns out that OpenSSH 7.0 seems to require that an “IdentityFile” parameter be expressly defined. I might be able to do this in ssh_config but instead I just created a ~/.ssh/config file with the line:

IdentityFile ~/.ssh/id_dsa_main

That got me farther. Now the error changed to:

debug1: Skipping ssh-dss key /home/tarus/.ssh/id_dsa_main - not in PubkeyAcceptedKeyTypes
debug1: Skipping ssh-dss key tarus@server1.sortova.com - not in PubkeyAcceptedKeyTypes

It seems the key I created back in 2001 is no longer considered secure. Since I didn’t want to go through the process of creating a new key just right now, I added another line to my ~/.ssh/config file:

IdentityFile ~/.ssh/id_dsa_main
PubkeyAcceptedKeyTypes=+ssh-dss

and now it works as expected. The weird part is that you would think this would be controlled on the server side, but the failure was coming from the client and thus I had to fix it on the laptop.

Now that it is installed and seems to be working, I haven’t really played around with Mint 18 much, so I may have to write another post soon. I do give them props for finally updating the default desktop wallpaper. I know the old wallpaper was traditional, but man was it dated.

This was a more complex upgrade than usual, and I don’t agree that you must base your system to do it, even from major release to major release. This isn’t Fedora. It’s based on Ubuntu which is based on Debian and I have rarely had issues with those upgrades. Usually you just change you repositories and then do “apt-get dist-upgrade”.

But … I might wait a week or two once they approve an upgrade procedure and let other people hit the bugs first, just in case. My desktops are more important to me than my laptop.

Hats off to the Mint team. I’m pretty tied to this operating system so I’m encouraged that it keeps moving forward as quickly as it does.