5 months into Manjaro: harmless yet unacceptable breakage
I installed Manjaro XFCE on my new HP ProDesk 400 G6 Desktop Mini PC on March 14, and it worked flawlessly and without any incident until today, August 25, when it broke. It shouldn’t have happened so.
OK, I know, I’m running Manjaro unstable, which is basically a customized Arch, automatically rebuilt by Manjaro with a few hours delay and no extra testing. But I had to satisfy some specific requirements:
- I wanted to be able to install the very last version of whatever piece of software I wanted, so it had to be Arch.
- I wanted it to have a neat installer, good defaults, and a tool allowing me to stick to a LTS kernel if I wanted to, because one thing I wanted above all: if something breaks, make it not be the kernel. (My problem with Fedora is that during the 6 months between a release and the next one, the kernel changes to whichever version is the latest one, and I did have breakages with that stupid choice of theirs. Officially, Fedora isn’t a rolling-release distro, yet it always updates the kernel to the latest untested version, with no way to keep tracking a still supported older version!)
- In the Arch realm, only EndeavourOS and Manjaro offer you an easy way to stick to an LTS kernel or to switch between kernel lines, with a major difference: EndeavourOS is only a set of scripts, settings, tools, and a few extra packages atop Arch, whereas Manjaro is a delayed rebuild of Arch with a different set of scripts, settings, and tools, with more consistent customizations, and many more extra packages. Therefore, Manjaro can’t use Arch’s repos directly, and it shouldn’t even use AUR, because most packages in Manjaro are at least 2-3 weeks older than in Arch; and yet, once you’re not using Manjaro stable, but Manjaro unstable, you’re practically using the same packages as in Arch, rebuilt with Manjaro’s settings where applicable, with something like a half day’s delay, and that makes AUR less risky to use in terms of compatibility with Arch.
- EndeavourOS is ugly and stupid, for they simply refuse (as a policy!) to provide any GUI package manager (say, Pamac, but I’m also using Octopi), despite their distro being so noobie-oriented! After all, distros like EndeavourOS do exist because Arch doesn’t have a decent installer, and EndeavourOS also has the online release notes and other documentation translated into a few dozens languages, yet they refuse to be of real help to the users by preinstalling the most demanded GUI tool of all, Pamac! Besides, no matter how much I wanted to like their project in spite of the logical inconsistencies of their approach, it really is ugly! I want an OS to be usable “as is,” without the need to fiddle with the default theme and other default settings; as it happens, XFCE or not, EndeavourOS lacks the elegance and the consistency of Manjaro, hence my qualifier: ugly.
- Now, most people should be happy with Manjaro stable as it is, for it has its own extra repo with a nice choice of packages (including e.g.
softmaker-office-2021), but I needed two things: (1) faster updates than those provided by Manjaro stable or Manjaro testing; and (2) maximum freedom, i.e. to use AUR. While the reliability of AUR is a hit-or-miss thing, once you know a PKGBUILD is good, you’re having a simple way of building and updating a package. If sometimes AUR’s PKGBUILDs are not in sync with the latest versions of the source code or of Arch’s packages, using AUR with Manjaro stable might lead to the opposite situation: a PKGBUILD is updated for newer libraries than available in Manjaro! That’s why I decided to go for Manjaro unstable, which still makes a better choice than vanilla Arch or most Arch derivatives.
And everything worked quite smoothly.
Until today when, after an update and a reboot, my PC entered the BIOS, because it couldn’t find anything to boot!
I was able to boot using either of Super Grub2 and SystemRescue (which I had on a Ventoy USB stick), so I knew the culprit was not the kernel (which, BTW, was still at 5.15.62-2-MANJARO, with 5.18.19-3-MANJARO as an alternative). So it had to be GRUB, which is the most retarded piece of software ever created on planet Earth! (Yes, I still miss LILO.)
I wanted to know what’s the root cause of the issue, other than the fact that GRUB was fucked up: was it Arch-only? Did it also affect Manjaro stable, or only Manjaro unstable, because of Arch?
The first report I found about this issue was on r/archlinux:
Latest Grub update – Testing Repo – couldnt boot
I am using the testing repos on both Laptop and Desktop PC.
Yesterday I did an update which had a Grub update (Grub 2:2.06.r322.gd9b4638c5-1).
Laptop worked fine but my Desktop PC was unable to boot. It was entering the BIOS menu since it was not able to boot.
The first time I recover the boot entries with Timeshift and it worked fine. Tried again to install the Grub update and again the Desktop PC failed to boot.
In order to fix this issue I have arch-chroot , reinstall efi , grub installation, mkinitcpio -p and everything worked.
It was a very weird issue , did anyone else had this issue?
Same thing happened to me now, right after grub update. Came here to check if this was a personal problem, guess not.
I had same today after grub update. Mine was on stable repos.
In the meantime, the thread grew longer.
Then, someone mentioned it in Manjaro’s forums, at the end of the thread [Unstable Update] 2022-08-19 – Kernels, LibreOffice, KDE Frameworks, AMDVLK, Qemu, KDE-git, Haskell, Python. Meanwhile, the issue was moved to a separate thread, Issues with GRUB on the unstable branch:
Same here means my machine boots into Bios after I updated grub package.
The boot into Bios happened with r322. The update was done ~3 hours ago.
Downside now is after going back I don’t have Grub selection anymore with my dual boot. I’ll fix that later. os-prober does detect my second boot option.
Edit: After a timeshift restore and a new install try of grub 2.06.r322 + all other packages it boots normal now.
What happened next?
Manjaro has reverted the
grub package from version 2.06.r322.gd9b4638c5-1 to version 2.06.r261.g2f4430cc0-1. This, however, only prevents further breakage, but it doesn’t repair the existing ones!
Because, you see, reinstalling the
grub package does NOT fix the issue; one has to reinstall GRUB in the boot partition!
Suggestions to fix GRUB? Here, here, or on Manjaro’s forums here. Since instead of using a Live Manjaro stick, I was able to boot into the Manjaro installation using Super Grub2, this is what I did in my case (no need to
sudo grub-install /dev/nvme0n1p1 sudo update-grub suro reboot
Any output of this kind:
/usr/bin/grub-probe: warning: unknown device type nvme0n1
is innocuous and can safely be ignored (as I said, GRUB is dumb).
All in all, it's fixed.
All in all, I took my chances when switching from Manjaro stable to Manjaro unstable five months ago, thus exposing myself to the risk of using Arch packages rebuilt without any relevant delay and without any testing by Manjaro.
All in all, whoever is using anything derived from Arch should know what they're doing. Such stuff is for power users.
But it's fucking unacceptable, because GRUB is such a fuckingly stupid yet essential shit (thanks, GNU!) that it shouldn't be changed without testing as if your life depends on it!
I would have accepted a broken 6.0.0rc2 kernel because, you see, it's not released yet, and even when it will, one shouldn't rely on it while it still says 6.0.0.
But to have GRUB break a system is simply intolerable, regardless of the distro and branch. How can I recommend Linux as a desktop (not as a fucking container) to anyone? No wonder some people who like throwing their money out of the window go for Apple's crap: even if nowadays their hardware isn't what it used to be, at least their OS boots all the time.
Fucking software developers. Aren't they always willing to change something that works, just because they can?
Well, unless I win the lottery (which I can't, because I'm not playing), I'll stick to Manjaro unstable. On the old laptop, I might however install Manjaro stable, without any use of AUR whatsoever. Just to have a more reliable backup machine.