I thought it’s going to be a slow, uneventful Monday. As it turned out, I had a lot to read. It all started from the first comment of today’s DWW. I stopped tilting at windmills, especially as systemd-free systems tend to have issues with some software, and I’m not thinking of GNOME right now. But some other people…

The issue at hand ■ Selected comments from The Reg (usually less technical) ■ Selected comments from Phoronix (usually more technical) ■ Luca Boccassi vs suspend-then-hibernate ■ BONUS: Wayland compared to systemd (or vice versa) ■ Can metastases be cured?

The issue at hand

That comment mentions this Slashdot story: Systemd 256.1 Addresses Complaint That ‘systemd-tmpfiles’ Could Unexpectedly Delete Your /home Directory.

Now, that’s quite a long read, with all the tree structure (tiens, arborescence existe même en anglais), so I won’t quote any selection of comments. OK, maybe this one:

The error of the user is to run systemd in the first place and to trust the cretins behind it.

Other mainstream media articles include: systemd 256.1: Now slightly less likely to delete /home (The Reg) and Systemd 256.1 Fixes “systemd-tmpfiles” Unexpectedly Deleting Your /home Directory (Phoronix).

🐞 It all started with this bug report: refuse systemd-tmpfiles –purge invocation without config file specified on cmdline #33349. A user of Debian sid, using systemd version 256, had this queer idea:

I noticed my /var/tmp directory grew quite a bit and wanted to manually clean it. Since I was about to log off for the day, I skimmed through the manual for systemd-tmpfiles(8).

–clean
If this command is passed, all files and directories with an age parameter configured will be cleaned up.

–remove
If this command is passed, the contents of directories marked with D or R, and files or directories themselves marked with r or R are removed unless an exclusive or shared BSD lock is taken on them (see flock(2)).

–purge
If this option is passed, all files and directories created by a tmpfiles.d/ entry will be deleted.

Added in version 256.

Not knowing much about the systemd-tmpfiles architecture, other than that it’s being used to clean up temp files, --purge seemed like a good idea. So, I ran sudo systemd-tmpfiles --purge, expecting it to clean my temp files.

This is how he lost a good portion of his home directory. Yeah.

Not helpful, Luca Boccassi, a systemd developer from Microsoft:

So an option that is literally documented as saying “all files and directories created by a tmpfiles.d/ entry will be deleted”, that you knew nothing about, sounded like a “good idea”? Did you even go and look what tmpfiles.d entries you had beforehand?

Maybe don’t just run random commands that you know nothing about, while ignoring what the documentation tells you? Just a thought eh

An asshole. Who would have expected that something in a place that’s supposed to relate TEMPORARY SHIT can actually delete $HOME?

We’ll get to that later. What I’ll post below is a selection of comments from the two aforementioned press articles. On occasion, I’ll comment on a few of them.

Selected comments from The Reg (usually less technical)

Thread:

Too complex!

This just supports my view that systemd has grown much larger than any of us ancient hacks originally imagined. In the beginning, it made sense to come up with a system that could replace maintaining all those init files. Now it has grown to be larger than Linux itself. Regrettably, it is getting much more difficult to find a non-systemd Linux distribution.
———————
Early systemd looks exactly like a copy of the Solaris SMF – which fixed a number of issues with the init system – dependencies, parallelism etc.

I recall there was discussion at Sun about whether it was too big a change to the traditional init system. Systemd took that big leap as a starting point and has spent the next 15 years trying to devour all of Unix.

I have no idea whether that was always the plan or it’s just a case of feature-itis.
———————
That’s the problem. You start out with a simple goal, let’s fix this stupid init file situation and take advantage of the multiple cores at our disposal to run things in parallel where possible.

So now you have something that has to manage a lot of dependencies, so it has a bit of complexity to it but you probably think you implemented that in a pretty slick way. You start seeing other things that done poorly that this slick system you set up could handle, so you add them, which which probably requires adding a bit more capability to your slick system, making it a bit slicker in your eyes. The more capability it gets, the more things it could “do better” which get brought under its control. Its like a black hole growing larger and larger as it swallows suns, giving it more gravity to influence space further away than before.

The black hole didn’t have a plan, it just grew based on what moved within its sphere of gravitational influence.
———————
“Its like a black hole growing larger and larger as it swallows suns”

Nah. It’s clearly a cancer.

Consider: systemd takes root in its host, eats massive quantities of resources as it grows, spreads unchecked into areas unrelated to the initial infection, refuses to die unless physically removed from the system, sometimes kills off important subsystems at seeming random, all the while doing absolutely nothing of benefit to the host. That sounds an awful lot like a cancer to me …

So do what I do and call it the systemd-cancer. Short, descriptive, accurate, has been known to scare management/moneybags away from distributions containing it … what’s not to like?

Thread:

Init marvellous

So now purging tmp files is the duty of an Init system, not, say, a cron job with the cleaner of your choice?[1]

Or am I just so totally out of touch that I’ve not realised that Init means “every single second of your system’s runtime, even when it is running stably” and not just that bit at the beginning with all the messages on the console?

[1] and far simpler ones than this beast, according to its manpage. Although, some of those options do look really useful, if you find that the rest of the Init keeps dying ‘cos the processes invoked keep messing up their own files/dirs and need them to be quickly recreated before said process is started. Not that anything in a systemd based setup would ever need such patching, would it?
———————
Not just tmp files, it feature creeped to do a bunch of other things but it’s still called tmpfiles because the maintainers can’t be bothered to change the name and until now the purge option still deleted everything it now manages as if they were just temporary files.
———————
It is probably fairer to say that systemd isn’t an init system. It contains one, but the project as a whole appears to be an effort to rewrite every Unix daemon from scratch and make the new versions the installation default for that particular purpose.

Viewed from that perspective, the obvious option is to use the init system or not according to taste, but avoid all the re-writes for a decade or two until they’ve had a chance to bed-in.

I don’t know if any distro has taken that approach. I haven’t heard of one. Perhaps it is impossible, in which case systemd’s own init process is clearly unfit for purpose.
———————
It’s not weird, it’s FOSS idealism functioning in the real world. Someone thought they could “do better”, just like a fork, and created a project to suit their own very personal agenda. Coding ensued.

The problem is that it grew, which really wasn’t a problem in regards to the FOSS belief; the problem is that it has been adopted by other systems without user community feedback or approval. The maintainers of the distros did a very unFOSS thing and imposed a system without option or choice.

What this shows is, really, the fundamental weaknesses of the ‘ideal’ of the FOSS system. The general public think they are being given a choice when choose FOSS but in reality you are simply getting someone ELSE’S premade choices to choose from, which is fundamentally not really different than any other system you can use. You have only more choices due to the absolute number of forks, created by every Tom Dick or Harry who thought they could “do better”. The choices are a bit of an illusion, the systems are simply an assembly of premade modules… that, mostly, are created by someone else.

So all these wonderful wizards from On High have decided that they like systemd, too bad for you that you don’t get a word in, [as usual] we really don’t CARE about your opinion (see: tmp –purge feedback. Because deleting files from /home that isn’t a “/home/temp” is ALWAYS a good thing…).

And that’s what an ‘unregulated’ FOSS system gets you – the Wild West of programmer’s ideas, ideals and beliefs with little oversight of someone looking over their shoulder and saying “Yes, that’s a STUPID idea”.
———————
Fair points, I’d also point to one of the core flaws of the team, which is a committed opinion that only THEIR opinions matter. They in this case being the community making code commits to THEIR project.

Not the Kernel team, not the commercial distros that pay for a big chunk of the work, not the universities that originally granted most of the code to the community and designed the core, the philosophy, and taught most them.

Not the hyperscalers or the server wrangling greybeards holding large parts of industry together.

So why not constantly introduce breaking changes all over the place, because none of those people matter if it runs fine on the programmers laptop and they like it better that way right?

The word be damned.

The only point of yours I disagree with is that the SystemD team in any way embodies “ideal FOSS development”. At this point they are a well documented embodiment of one of some of the oldest anti-patterns. SystemD should be taught in grade school and one of the cannonical examples of the blob. The people that made this mess should be forced to break it back up into pieces, and further attempts to spread the cancer to additional functions should be block from the upstream distros till they get their sh*t together.

Sadly, the community lacks collective organization or internal accountability structures to address out of control Devs and Dev teams like this. Linus basically kicked them out of the Kernel some time ago. His leadership model probably isn’t a solution for the big distros, but something that gets the same job done is needed outside the kernel team just as badly.
———————
My response is “Exactly”, to both of you. Jake is blind to problems of FOSS as an apparent True Believer. FOSS can be great… but it can also be an unmitigated disaster, and sometimes the difference is only based upon which developer is doing what. With little to no community feedback on some projects – or, even worse, a complete dismissal of user feedback because of ego or other issues – you can end up with useful FOSS, or simply a /home-deleting mess. Two decades of “But so many eyes are on the code! FOSS can’t fail!” yet plenty of bugs and security blunders are recognized now.

It is NOT a guaranteed fix it. FOSS has benefits… FOSS has problems. Grow up and acknowledge this so you move forward rather than putting your head into the sands of denial.

This one:

So, the initial response from the systemd maintainers was…

…essentially, “works for me. #WONTFIX.”

Plus ça change…

Thread:

Did anyone of us actually ask for systemd?

And why did other distributions so readily adopt it?

25 years ago we nicknamed Suse “The Windows from Nuremberg”, and Redhat was being referred to as a ‘job creation program’ (not a bad thing in itself, but): ‘for anyone who shouldn’t be let anywhere near a keyboard’. It begat a new cohort of Linux ‘experts’. And don’t get me started about the Debianauts or the Ubuntans.

Short of rolling your own (if you have the time), Slackware is the only one I can think of which still leaves you in control. (But there you have to actually know what you’re doing – and if you do, it makes for happy users, and also means that when your phone rings, it is only for a change request.)

(Just in case: I’ve worked – not just used – with Minix, Coherent, A/UX, Irix(MIPS), Almos tImitating uniX,SunOs4.1 and upwards, various BSD flavours, and of course, various Linux distributions (where I still have fond memories of Yggdrasil – the sole reason I learned Tcl/Tk))..
———————
In a word: Gnome.

Gnome these days depends on SystemD. There’s a lot of RedHat controlled resource working on Gnome and systemD, and they’ve seen fit to create that dependency. If you want your distro to have Gnome as an option, you have to have systemD. And unless you’re totally stark raving mad, your server spin of your distro is going to be much like your desktop version. So, that gets systemD too.

It’s interesting to consider if RedHat has any ulterior motive for bending all Linux distros this way, by using it’s control over dev resources to out-code competitors. What might RedHat actually do? Well, having got the Linux world irrevocably stuck with systemD, they could decide to stop making their copy of the source code publicly available (just like they’ve done with RHEL). The result is pretty bad for other distros, if RedHat chooses to internally fork these things, and emits only binaries to paying customers.The public fork falls behind pretty quickly. The combination with RHEL now being (effectively) closed source is quite severe too. Basically, if IBM bought Ubuntu too and pulled the same trick, then Linux = IBM, and (effectively) closed source. Would global competition authorities step in and prevent such an acquisition? I doubt it.

Oh there may well still be a Linux kernel project publishing their source as they do today, but if the two major branches of Linux fall under IBM’s control then effectively they get to choose what kernel users are actually able to run, and it won’t be free.
———————
One problem with your cunning plan: According to Linus himself, Linux (the kernel) is not now, and never will be, dependent on the systemd-cancer. And both the SysV and BSD inits are freely available to bolt onto the Linux kernel. And the entire world of GNU tools are available, none of which require the systemd-cancer to work properly. And I’m pretty certain x.org will never mandate the systemd-cancer.

In other words, there will always be a way to roll out a Linux distro that will be free of the systemd-cancer (and GNOME. (And Wayland, which you somehow forgot.))

Or, you could just run Slackware.
———————
Poettering and the rest of satan’s little helpers have a completely different point of view and are hell-bent on making it prevail. No matter. As the systemd cancer spreads, it’s inevitable its crufty bloatware will soon assimilate the kernel. Assuming it hasn’t already done that.
———————
Both systemd and gnome can go fsck themselves, neither are fit for purpose.
———————
The LSB decided that systemd would become part of it. Just before it quietly disappeared – now even the LSBs stearing comittee’s site contains only a 404.

Good question:

What’s it Actually For?

I know I’m asking a stupid question in asking, “what is systemd-tmpfile actually for?”, like I’m expecting some logic.

The suggestion I’ve gleaned from the man page for both it and its config file format is that you’d use it as a general purpose file system environment creator for some service or other. E.g. if a service wanted a directory and a named pipe in it, you’d create a configuration for systemd-tmpfile to create those and systemD would be able to clean them up afterwards.

Except that I’m left wondering why on earth one would ever want to create the prerequisites that way in the first place. Why wouldn’t the service simply be coded up to create what it wants for itself, and to clean up what it no longer needs? If, as a service developer, I decide “no, I’m going to use systemD for this filesystem environment creation purpose”, I’m now needlessly and (so it would seem) complicatedly dependent on the behaviour of the tool. That’s a bad thing because it seems the behaviour is still evolving (hence, people getting caught by surprise). And, my service’s file system environment can now be tinkered with by the system admin in ways that could be wrong preventing the service from operating properly in the first place. None of those things can go wrong if my service does all this creation work for itself, taking no more than a single parameter for the path where the user wants all these file system objects to be created.

I just don’t see the point. Anyone got any believable reasons why one would actually want to use it?
———————
Because the systemd cargo-cult knows best. So just suck it up and don’t ask awkward questions.

Here! Drink more of the systemd kool-aid. It’s good for you, rich in the finest unmaintainable bloatware.
———————
The rationale goes like this:

developer of service xyz (systemd unit), which needs “a directory and a named pipe in it” uses its own way to create temp dir like xyz_/ but that may not work in some obscure context, maybe in a container or whatever.

The missive from systemd HQ is to rely on systemd-tmpfile instead where systemd guarantees it works for you. The consumer of tempfiles does not have to deal with edge cases any more. Overzealous –purge notwithstanding.

Again:

Why would anything in /home be in tmpfiles.d?

Are they implying that the people who packaged the OS are at fault, since tmpfiles.d shouldn’t have entries for things that aren’t, you know, temporary files, like one’s home? Or are they reminding us that life is fleeting and that nothing is really ever permanent?

By their logic, it wouldn’t be unexpected if running “rm” with no options removes everything in the current directory… Not sure I like that thinking.
———————
The f’ing tool should default to doing nothing.

Anything else is just asking for trouble.
———————
Are you referring to systemd here, or Poettering/Bocassi?

I mean, either way, you’re right…
———————
I wondered that too since as a Debian dev I saw the initial reports of this on IRC and initially thought it was due to some Debian specific packaging, but it turns out this config was added upstream NINE years ago in 2015. Reading between the lines of the commit messages I suspect this is part of the systemd focus on creating/generating, and starting the host with immutable full disk images that include partition table and all file-systems.

Looking at the 2 commits it looks like the original intention was simply to ensure these directories are present on boot, but a side-effect of a –purge is they’re also all cleared! Not nice that it could include remote file-system mounts since this specific file includes /srv/ but there’s other vital directories listed in the commits (`/var/ anyone ?)

systemd$ git l tmpfiles.d/home.conf
…
…
Author: Lennart Poettering lennart@poettering.net
Date: Wed Oct 21 19:47:28 2015 +0200
tmpfiles.d: change all subvolumes to use quota
Let’s make sure the subvolumes we create fit into a sensible definition
of a quota tree.
diff –git a/tmpfiles.d/home.conf b/tmpfiles.d/home.conf
…

A long(ish) comment:

From what I’ve learned, Gnome can use (reddit link) either systemd-logind, or Consolekit (or at least, it did so 11 years ago). ConsoleKit was reported dead 11 years ago, the last maintainer being one L. Poettering.

Gnome apparently uses various systemD utilites for configuration.

Three years ago it was reported that Gnome builds on FreeBSD, except that FreeBSD (another reddit link) provides some patches that allows Gnome to be built without the systemD libraries. The last update seems to be more than 2 years ago, which would be a bit odd if Gnome could simply be built unpatched on any common or garden *nix system.

Ten months ago I’ve seen reports that, yes, Gnome ran on Devuan, but the small handicap was that (yet another reddit link) nothing could be run, not Firefox, nor Brave.

The Wikipedia article on Gnome says that Poettering himself wanted to make systemD a dependency, which is probably where the fuss all started. And indeed it seems there is a deliberate policy of full-fat Gnome being dependent on it. For example, there’s no doubt a *nix way in which Gnome could have supported multiseat configurations, but no, they’ve gone and relied on systemD for that.

Ok, so that’s a bunch of unreliable reddit links and a more reliably cited Wikipedia article. However, it seems far from safe to say that “Gnome runs on all systemD free distros”. Patched Gnome evidently does, as would stock Gnome atop reimplementations of bits of systemD that it wants (hence, elogind…), and the Gnome project has anyway set out to provide only basic functionality without systemD. This is a somewhat removed from what other desktop projects do, which is to be pretty much *nix OS neutral.

Gnome 2 predates SystemD by 8 years, and Solaris had Gnome 2 long before systemD came along. I used to use that (ah, the memories). Solaris running Gnome 2 is in no way an indicator that the BSDs can trivially build and run Gnome 40+ in the modern era.

Regardless, as a means of influencing other distributions, Gnome is a potent tool. RedHat are the largest contributor to the project, it’s one of the more monied projects, and it can probably out-develop all other OSS desktop projects. If RedHat wanted to they could drag the project off in a direction that makes it even harder for non-systemD OSes. This is isn’t a matter of “they’d never do that”; RedHat are a confirmed, profits-at-any-cost corporation, clearly with no intention of sticking by the accepted norms of OSS where it really counts. They’ve gone from being a company making money from support to a company making money selling binaries built from what is largely other people’s source code. If they can get commercial advantage by manipulating Gnome or SystemD, they probably will.

From the point of view of other distro projects that wish to survive long term, relying on RedHat playing “nicely” seems ill-advised.

Selected comments from Phoronix (usually more technical)

#2:

god, i hate how this mastodontic piece of shit is basically mandatory now

#18:

Why is no one actually addressing the real problem this issue highlights:
“Including the /home that is created by systemd-tmpfiles’ home.conf. “

What kind of stupidity is this?
Why are a function or command to create temp-files used to create /home? WTF?

Is this done by a distribution or bad system administrators, or is this intended or recommended way from the systemd developers/projects?

If it’s the first, a note on the specific distro had been nice. To avoid something made by incompetents.

If it’s the second, the systemd developers are even worse than I thought and they deserves much worse than he comments in this tread.
That is beyond stupid!

#27:

Systemd walks into a bar, shoots the owner, proclaims it is the new owner, turns the bar into a casino, adds a hotel, a grocery store and an init system.​

#28:

I like systemd overall, but some of the devs are real jackasses. The tone and wording of that response reminds me of the systemd sleep.conf change where the dev basically told everyone to fuck off and they were using it wrong, even though the sleep then hibernate usage was an obvious and hugely useful use case for s2idle laptops.

I’ll discuss the hibernation issue below.

#29:

suspend-then-hibernate: add option to force early hibernate · Issue #25269 · systemd/systemd

refuse systemd-tmpfiles –purge invocation without config file specified on cmdline · Issue #33349 · systemd/systemd

systemd developer Luca Boccassi (“bluca”) of Microsoft

The inane responses by “bluca” almost make it seem like Microsoft is making systemd bad on purpose to sabotage Linux…

#30:

Ahh…so it’s the same jackass from the sleep then hibernate thread. I guess it’s good that there aren’t two of these yahoos. But maybe this guy shouldn’t have anything to do with triaging bugs Microsoft.

A useful explanation at #33:

systemd-tmpfiles would have already long been renamed (there were talks about it tmpfiles: naming is inconvenient #12447), but it would break scripts and would just lead to more bikeshedding about what name would be appropriate (it’s systemd-sysfiles in every case).

The reason that home.conf exists is because systemd is able to set up an entire system with only a /usr/ partition and an empty root partition (or tmpfs). It is used to make /home/ and /srv/ a subvolume instead of a regular directory on btrfs.

#52:

It should have been obvious – your /home is just a temporary fluff compared to the might of systemd.

#55:

The average user would not run systemd-tmpfiles –purge (an option that was introduced in 256 and now has guardrails in 256.1). The average user is much more likely to use the french option for rm on their home dir.

However, the most problematic part in this whole story is Boccassi’s attitude.

Another useful explanation at #64:

As always the comments on these kinds of threads turn into unhinged theories and a vessel to channel people’s rage towards their favorite pet peeve topics. Rather than, y’know, asking why distros use systemd-tmpfiles for generating /home directories in the first place.

For automation, containers, VMs, test environments etc. The rule in question creates a /home subvolume with its own quota group if the dir/subvolume doesn’t already exist.

Edit: tmpfiles allows you to describe which files directories should exist on a system and what attributes they should have in a declarative way that is composable/extendable and is not a shell script with a bunch of touch/mkdir/chown/chattr/chacl/find/rm.

As expected, the Big Designers of the Linux ecosystem couldn’t give a dead rat’s ass on Linux on the Desktop™. All they care about is servers, VMs, containers, everything else but your desktop. Well played into Microsoft’s and Apple’s field.

A grumpy boomer at #75:

I don’t trust the “intelligence” of open source “developers”, nor their “integrity”.

The best lies are the ones that have a decent portion of truth in them.

The best scams and hoaxes are the ones that are done right in the open.

The best thieves steal while you are looking at them and convince you to help them,

Open source is a scam and always has been,

The first programming class i ever took was more than 40 years ago, I took a class in BASIC and I remember the first day the instructor told us the most important thing is to always comment your code, because no one is going to know what you are trying to do and you are not going to remember if you have to revisit it down the line.

If you want to see how half-assed open source development is and how big a scam it is, here is an interview with Linus where he is talking about the lack of kernel documentation: https://www.youtube.com/watch?v=bAop_8l6_cI&t=2275s

This is the garbage that you and other put their trust in.

I corrected Linux into Linus above, but go to that video from Linux.conf.au 2015 (Auckland, New Zealand), at minute 38. You’ll see Linus Torvalds very relaxed about the fact that no one understands how the kernel works and that only some major concepts are documented. For the rest, he’s an adept of … biology and evolution! I’m not kidding you.

This being said, the author of the comment above has been severely trashed in subsequent comments. People prefer to live in denial. Linux is the lesser evil, not the Heaven, dammit!

Skipping the reactions to the above comment, #76:

Oh wow. I’ve seen plenty of stupid shit in my life, and I’ve been dealing with *NIXes in one way or the other since the early 90s. But this one is truly flabbergasting.
Seriously? Wiping out the user’s home? Please don’t tell me they’re trying to assign the blame to the users for not checking what files were about to be deleted.
The lack of professionalism and the systemic antagonism towers the systemd userbase is a toxic and persistent problem. Regardless of systemd’s merits, or the lack of them.
Bravo, just bravo. What next? Discarding responsibility by saying in this day and age everyone should have a backup?
Proselytizing on the need of yet another task, responsibility, and layer of complexity by proposing some mechanism to encapsulate your home into an encrypted monolith that can be made portable everywhere?
I’m looking forward to seeing these clowns start dabbling in encryption. If you think wiping out your home was fun, just wait until then they tell you what happened to the encryption keys…

#82:

Fuck this shit, I’m out. I don’t want to run Linux on any desktop ever again, I’m done.

Not overreacting, eh?

#100:

systemd-microsoftdevs –purge isn’t in the manual at all, but they should run it anyway. For science.

#128:

Nothing ever runs systemd-tmpfiles --purge, which this article is about. Just like nothing ever runs rm -rf --no-preserve-root /. systemd-tmpfiles --purge didn’t exist before v256 released two weeks ago. Who are the people and how many of them, who lost their data by invoking systemd-tmpfiles --purge manually?

#131:

Why am I not surprised by Luca Boccassi’s response – he is a systemd developer.

#136:

Just checking: are you aware of systemd being free software and that you can read its source code in full?

Oh, but why only reading systemd‘s source code? Let’s read the source code of every single package that has been updated! To quote from another article in The Reg:

Debian 12 … is 1,341,564,204 lines of code. That’s the project’s own estimate. One and a third billion, that is, one and a third thousand million lines of code. For comparison, Google Chrome is about 40 million lines, which is in the same ballpark as the Linux kernel these days.

Nobody can read the source code of Chrome. Not alone, not as a team. Humans don’t live long enough. Any group that claims to have gone through the code and de-Googlized it is lying: all that’s possible to do is some searches, and try to measure what traffic it emits. A thousand people working for a decade couldn’t read the entire thing. …

We consider this normal. Everything is like that. It’s just how it is. …

The world runs on software, produced and consumed on an industrial scale, always getting bigger and more complicated.

Nobody understands it any more. Nobody can. It’s too big. But it’s the only model of making and selling software we have, so we are trapped in it.

Luca Boccassi vs suspend-then-hibernate

🐞 There has been talk about Signor Bocassi’s behavior regarding the bug suspend-then-hibernate: add option to force early hibernate #25269.

The issue was opened on Nov. 5, 2022, for Debian 12:

Since 252 I can no longer suspend-then-hibernate using systemctl suspend-then-hibernate.

If I execute the commands to suspend only it does work, same for hibernate. But not in combination.

UPDATE: I tested on Manjaro which is using systemd 251.7 and is working fine.

OP:

When executing suspend-then-hibernate it will go into Suspend mode but it will not follow into Hibernate after. This is what my login.conf and sleep.conf look like.

[Login]
HandleLidSwitch=suspend-then-hibernate

[Sleep]
HibernateDelaySec=300s

I checked dmesg but it doesn’t show any errors.

Signor Bocassi:

That’s expected, see the manpage. Hibernate will happen when low on battery.

OP:

I think you are miss understanding my issue. I did have it working before systemd 252.

Signor Bocassi:

https://www.freedesktop.org/software/systemd/man/systemd-sleep.conf.html#suspend-then-hibernate

Signor Bocassi:

bluca closed this as not planned on Nov 5. 2022

For reference, the document linked to has it extremely clearly:

suspend-then-hibernate

A low power state where the system is initially suspended (the state is stored in RAM). When the battery level is too low (less than 5%) or a certain timespan has passed, whichever happens first, the system is automatically woken up and then hibernated. This establishes a balance between speed and safety.

If the system has no battery, it would be hibernated after HibernateDelaySec= has passed. If not set, then defaults to “2h“.

If the system has battery and HibernateDelaySec= is not set, low-battery alarms (ACPI _BTP) are tried first for detecting battery percentage and wake up the system for hibernation. If not available, or HibernateDelaySec= is set, the system would regularly wake up to check the time and detect the battery percentage/discharging rate. The rate is used to schedule the next detection. If that is also not available, SuspendEstimationSec= is used as last resort.

Added in version 239.

So, as long as HibernateDelaySec is set (and it’s set to 5 minutes), should this happen before the battery reaches 5%, the system should be unsuspended, then hibernated! And this is what happened before systemd version 252.

  • If the system has no battery, i.e. it’s a PC, not a laptop, it will be hibernated after HibernateDelaySec or, if not set, after 2h, because it will never reach 5%, being at 100% at all times.
  • If the system has a battery and HibernateDelaySec is not set, no default of 2h should be considered. Instead, only the battery percentage will matter. But if HibernateDelaySec is set, the system will regularly check both the time and the battery, to determine which condition occurs first: in our case, 5 minutes (300s) or 5%.

That was an explanation for the mentally retarded Lucca Bocassi from Microsoft, who can’t even read a page!

Another retard, TriMoon, a minion of Sig. Bocassi, rushes in without properly reading the documentation:

Maybe the functionality was broken in v252 and is now fixed 🤔
The documentation @bluca linked to clearly states that hibernation will only happen if the system is woken via the ACPI low-battery (<5%) signal.
Otherwise suspend-then-hibernate will behave just like suspend only.
So you might need to change your settings to do a hibernate instead.

Or maybe you’re an ass.

Another guy, Nicolas Peugnet, pops in with an interesting piece of information:

Well this is a little bit easy to refer to the latest docs which changed since v252.

On v251 the manual entry for this option was:

suspend-then-hibernate
    A low power state where the system is initially suspended (the state is stored in RAM).
    If not interrupted within the delay specified by HibernateDelaySec=, the system will be woken using an RTC alarm and hibernated (the state is then stored on disk).

Why did this change instead of creating another power saving mode (for instance suspend-then-hibernate-on-low-battery)?
How to use the previous behaviour?

I dont want my computer to stay in S3 state when it is >5% battery. I want it to go to S4 state after HibernateDelaySec of S3 state.

S3 is Standby (suspend-to-ram); S4 is Hibernate (suspend-to-disk). To me, regardless of the different wording in 252 vs. 251, the behavior should be the same. If the person who changed the documentation had a different behavior in mind, the resulting text does not convey it. I’ve read a lot of legalese in three languages, and I can corroborate the statements in three paragraphs, conjunctions and all included. I also used to develop software, if that matters.

The new version of the documentation does not describe a new behavior; it just uses an extremely convoluted way to describe the old one. Make this experiment: find a good lawyer and make him or her decide on what the described behavior means for a laptop with a defined timeout. In the worst-case scenario, “it depends on how a court of law would interpret it, as it’s not entirely unambiguous.”

Even so, a judge would normally consider the intended behavior, including the rationale for the previous one.

I’m however surprised that someone at systemd, anyone, had bothered to adjust a behavior that includes hibernation. Hibernation is so much disparaged in the Linux world in the last ~15 years, that many distros just don’t care about it. I’ve been called an idiot by various Gen Z pieces of shit, because “it’s obvious that suspend-to-ram is enough.” It’s fucking not.

Signor Bocassi:

That is not supported anymore, as it was half-broken and made very little sense, it was just a workaround for missing functionality that is now implemented. If you want to hibernate it immediately, then just do that.

I’d rather say that Sig. Bocassi makes little sense. What workaround? What missing functionality? Delayed hibernation based on two criteria—power and time (whichever condition happens first)—did work before!

Nicolas Peugnet:

If you want to hibernate it immediately, then just do that.

I don’t want to hibernate. Resume from hibernation is around 45s on my system while resume from suspend is instantaneous.

I want to always suspend, then after 30m hibernate. If I need to quickly reuse my computer after I suspended it, it resumes instantaneously. And if I need to use it later, then I can probably wait 45s, so I want it to hibernate to save power.

Signor Bocassi:

Why 30 minutes and not, say, 4 hours?

Nicolas Peugnet:

Why is it important? This is my setup, which was previously configurable with HibernateDelaySec. I happen to like the value of 30m (in fact I just checked and I used 45m, but I don’t see the point of this question).

Signor Bocassi:

The current behaviour is fully expected and documented.

Yu Watanabe, from Tokyo:

Hmm, but as reported, v252 changes the behavior. That’s true regardless of it is expected or not.

How about to honor the setting if specified instead of using ACPI_BTP or calculated discharging rate?
And/or, introduce another knob(s) to control the interval for calculating discharging rate?

Anyway, please do not close this page so quickly.

Signor Bocassi:

Hmm, but as reported, v252 changes the behavior. That’s true regardless of it is expected or not.

Yes, and it was a 6-months long project, widely advertised and listed in the changelog as expected. The purpose of this functionality is to avoid power getting cut and data loss, so the previous implementation was just a bandaid, and a bad one at that, and it is not something we want to support because there’s just no way to guarantee that there will be no data loss or worse.

The use case of “I happen to like the value of 30m” is not sensible and definitely not something to support.

It’s hard to understand what was the purpose of that 6-month project, and where was it widely advertised! The previous behavior was just fine, and any improvement should not destroy what worked before! (Regressions are the rule of law in Linux.)

Nicolas Peugnet wouldn’t have it:

The purpose of this functionality is to avoid power getting cut and data loss

Once again, this is the purpose now and I think it is clear to everyone that you prefer it over the previous one.

so the previous implementation was just a bandaid, and a bad one at that, and it is not something we want to support because there’s just no way to guarantee that there will be no data loss or worse.

The previous purpose of this feature was to go into hibernation after a period of time. Which can be used to save power and battery life. The current behavior does not allow that so it is not just better.

The use case of “I happen to like the value of 30m” is not sensible and definitely not something to support.

At least accept that this is a matter of preference, as you prefer to have your battery drawn untils it’s 5% before it goes to hibernation. Whereas I prefer my computer to go into hybernation mode after a period of time to save battery.

First I have never experienced any loss of data as 30min is enough for the battery to survive in suspend mode even at 5%. But most importantly, I don’t see why it would not be possible to have both in suspend-then-hibernate mode. I.e. have the computer to go into hibernation after a period of time and if it goes below 5%.

You can even set the value of HibernateDelaySec to infinite by default if you prefer, I don’t care, as this is a parameter I can change. But please consider the usecases of other people than you.

I’ll skip a lot of the extra talk. A quick one from Felix Meyer-Wolters:

I think it is better to make HibernateDelaySeconds= specify the maximum amount of suspend time to keep backward compatibility.

Yes, please. This breaking change cost me countless hours and pain over the last few days, because I was debugging a dead end: https://bbs.archlinux.org/viewtopic.php?id=281373

I also miss the old behaviour of HibernateDelaySeconds…

I’ll skip some more comments. Enter Diana:

This is a serious security regression, as anyone who was relying on systemd to do what it said it would until version 252 is now potentially vulnerable. I need my (fully encrypted, verified boot) system to hibernate when I tell it to for security reasons, and until recently it did. And now it doesnt.

This is an extremely serious vulnerability and its baffling theres been a month of “debate” on whether a silent, backwards incompatible change, to an existing feature and option, that causes real world security issues is a problem or not.

Before anyone gets into it: Your threat model is not my threat model. This is within mine, and i’m sure also within plenty of organizations.

Not to mention the basic usability issues, I for one like to come back to my laptop and resume where I left off with my battery where I left it, not empty or at 5%.

Johannes Burk:

I would not name it a serious security regression. You can still just hibernate your system to secure the sleep state with full disk encryption. And I wouldn’t rely too much on the interrupt for hibernate after a delay when your security concerns are strong. 😉

Meanwhile, anyone who can’t wait for the feature to return and isn’t able to downgrade can work around the regression by using a custom suspend-then-hibernate script.

Diana:

Just because it can be worked around doesn’t mean it isn’t a regression. The behavior worked and was secure, and an update made it not. That’s called a security regression.

Now if everyone who was using it wants that same security, they have to first know about this issue somehow(CVE?), and then manually work around it until the proper fix comes out.

Signor Bocassi:

Given trolling has started, time to lock.

BONUS: Wayland compared to systemd (or vice versa)

When on November 27, 2023, Red Hat announced their Red Hat Enterprise Linux 10 plans for Wayland and Xorg server, Liam Proven wrote in The Reg: Wayland takes the wheel as Red Hat bids farewell to X.org. Comments ensued. Here’s a selection:

Wayland itself is getting old now…

What’s the betting that someone starts developing a replacement for Wayland sometime soon?
———————
So little to show for 17 years’ effort. Why are we trusting these losers?
———————
Don’t you worry, systemd will have a replacement for X in the April release.
———————
systemd will probably contain linux before long.

…

Wayland support is coming to the Mate desktop

Which is all that I really care about.
———————
I’d be up the proverbial creek without a paddle if it weren’t for MATE but it drives me mad that RedHat’s sabotage of Linux desktops has now leaked into lots of parts of MATE, e.g. the disk manager and “Simple Scan” which are GNOME applications and use client-side decorations and the horrendous buttons and hamburger menus that get embedded into the fat not-quite-a-title-bar.

RedHat has wildly trashed a lifetime’s worth of UI concepts and forced it onto just about everybody, removing all the flexible and user choice which we once had. So I’m at the point if thinking that even if Wayland offered me anything that X doesn’t, it’s got RedHat stamped all over it so I’m out.

MATE on the other hand is run by an awesome bunch of amazing people who continue to deliver worthwhile features and improvements. I really hope that a similarly amazing group of people and companies come together soon to stick two fingers up at RedHat and work either to keep X11 alive or implement something like the X12 idea with sufficient backward compatibility.

Meanwhile I’m going to figure out how to obtain the latest release under Ubuntu 20.04 to get new feature to disable mouse scrolling in the window selector and check out whether Pluma has any kind of automatic recovery yet.

…

I wish it wasn’t such a mess

Whilst Wayland basically works (for various definition of works) it’s very sad that Wayland is so very obviously oriented towards commercial interests and Linux first, and X for all its many many faults generally worked cross platform and had agreed on some functionality that the vast majority of Xorg implementations feature.

This is written on a fanless FreeBSD system with built in basic Intel GPU running Wayland and the labwc compositor. Firefox works fine, as do OpenOffice, WINE, and various Wayland applications for the most part.

There are still many issues, largely down to the ‘open’ nature of Unix standards, and the Linux first (let’s be honest : anything else very grudgingly considered) attitude to Wayland development.

labwc has an annoying ‘snap to edge’ feature if windows are placed in a particular position. There’s probably a way around this, but even if I find it, it may be different on every other compositor.

labwc does feature various Wayland protocol enhancements. Other compositors do not, because even if they’re based on a common compositor library, protocol updates in that library do not automatically flow through to updates in the compositor.

Leading from the above, you can’t rely on something such a config file to position monitors on startup (as you can in Xorg) as it may not be supported by the compositor, and the syntax differs for each compositor that supports it. Once Wayland itself is fully running, a post startup script can call the protocol enhancement to move monitors, and that is compositor agnostic assuming the protocol is supported.

The reference Wayland compositor, Weston, IS LINUX ONLY. There’s an old NetBSD port last time I looked, but the only up to date implementation is on Linux. The fact the reference compositor is not tested on a non Linux platform beggars belief.

It’s still not clear to me how to position windows that use the overlay plane (such as clocks). I can start the programs, but then can’t easily interact with them, because my compositor appears to have no built in support or documentation on this, it’s not obvious via a search, and at that point I start to lose the will to live and just look at an actual physical clock instead.

17 years on, things like colour management are still not nailed down.

Driver support for older cards is lacking, particularly on Nvidia.

What it needs, and also what will never happen :

ONE compositor base that everything bolts on to. This must be a suitable base for all compositor addons from the largest commercial desktops to the smallest individual offering.

Cross platform first, or at the very least a way of gracefully handling a lack of functionality. If it doesn’t built on Linux and BSD, it doesn’t get released.

Development driven by design, not commercial interests. So Redhat/Canonical, or Nvidia/AMD/Intel can’t move things towards ‘oh, your graphics card is old, no support for you’ [1]

As it is, things will coalesce around a small number of large desktop based compositors (i.e. GNOME), and any company with sense will insist on their open source supporting software running on : Linux (possibly x64 only), and GNOME Wayland. And nothing else. mmmmm open platforms, and choice. [2]

[1] To be fair this already happened multiple times in Xorg, it’s just that most people didn’t notice because drivers were re-written to support the new X interface (there are at least three I know of). However, Wayland seems potentially to offer a larger excuse and ‘not all the drivers are open, and doing this costs money, so even though your card is easily capable of working with Wayland, it never will’.

[2] Also to be fair, this occurs with poorly written Xorg software also : ‘what do you mean my function call to place an icon on the desktop or query desktop stuff is failing, because you’re running a window manager with no desktop or icons’.
———————
“Driver support for older cards is lacking, particularly on Nvidia”

Not really the fault of Wayland to be fair. NVIDIA seems to have a deep seated hatred of playing on a level playing field. If something exists that supports any cards and their cards look bad on it, they won’t support it or provide any compatibility for it.
———————
Technically you’re correct, but in this case it’s not the best type of correct. If not Wayland, then who? It’s yet another human problem where technically it’s not the problem of a specific person, but if the community as a whole wants a thing to be a success, then they need to do it anyway.

Just like Microsoft was forced to write some drivers for hardware in versions of Windows where the driver model changed and the manufacturers refused to update the drivers, because the hardware was common and it would have led to bad publicity. The question of course is ‘who pays’.

I may be off base here, but I don’t think the open source community (and many others) are prepared to pay someone for the drudgery to make things like Wayland a success. A lot of it is thankless, irritating, difficult work that also requires significant co-ordination and people skills.

When Linux and BSD started out they had the advantage of being able to build upon years of compromise between Unix vendors. Now commercial Unix is largely dead, and new technologies are basically Linux.

Technically BSD works too, but this is only because the BSDs added sufficient compatibility layers so BSD looks enough like Linux to make porting drivers easier.

Compositors work on both Windows and Mac OS because Microsoft and Apple have paid several people enough money to code and test the infrastructure until it works and they can enforce their own standards. There’s then a reliable infrastructure, generally driven by one large company, that everyone adapts around.

The same is happening in Linux, except it serves the interests of a couple of large commercial Linux vendors and is not oriented around the open source ethos. So what is resulting is technically called ‘Linux’, and things such as Wayland may technically be ‘open source’ and ‘cross platform’ but the reality is a long, long, way from the Unix ideals and intent in 1992.

If anyone has a more accurate view of the above, I’d appreciate some input.

I think the issue is that this is also exactly what some modern Linux users want : something free that is gradually becoming far closer to Windows than they’re happy to admit, and damn anyone else.
———————
“I think the issue is that this is also exactly what some modern Linux users want”

I think quite the opposite. All I want is for hardware vendors to allow support for open APIs rather than enforce their own proprietary bullshit on software developers. AMD, which is a vastly smaller business than NVIDIA contributes a lot to open source and open standards in general, as does Intel. Wayland works flawlessly on AMD cards…it does on Intel cards as well…but NVIDIA…nope!

NVIDIA just kinda looks fucking lazy or incompetent… I’ve got a pretty long memory, and it wasn’t all that long ago that NVIDIA drivers were just crap in general. Not just on Linux.

The day NVIDIA gets knocked off it’s perch is a decisive manner, is the day they will never come back…unless they change their ways and show consumers that they do in fact give a shit about something.

NVIDIA may make extremely performant kit, but the whole organisation…right up to that leather jacket wearing wanker…feels incredibly phony. I bet he can’t wait to get that naff looking jacket off when his PR people aren’t looking.

I’m always suspicious of CEOs that dress like peasants or truckers…because you know they’ve been told to dress like that for PR reasons, “it makes you seem more approachable / relatable / like you don’t give a fuck etc etc…and if they can’t decide how to dress themselves, how the fuck can they make any other decisions?

The last thing we need in a CEO is someone that isn’t human that wants to appear human…because it weirdly makes them even less human.
———————
Except I’ve had nvidia drivers running fine on X for over a decade, they’re kept up to date for new hardware. So I don’t think we can necessarily conclude nvidia are the problem here.

…

Integrate Wayland into systemd

Problems solved.

…

“Oh that issue has been fully resolved in Wayland!”

Said Nobody, ever.

…

I wouldn’t care if it worked

Still using X on my Fedora desktop due to desktop incompatibilities and difficulties with QHD display. Tried to get the default Wayland to work, failed, changed back to X. Killing simple remote application running (x forwarding and x2go) is another genius move. Are they fixed yet? Maybe, sadly I’ve been hearing for years “not a priority” or “not a use case” and I’ve stopped expecting it to ever work, or the developers to ever be interested in the features their users need.

Middle mouse paste, another thing they removed and were refusing to implement, looks like it is now there (the similarity to Gnome changes that were forced on users as “good for us” for a couple of releases before they changed their minds has not passed me by), does it use a primary buffer (last selected text)? Who knows without installing and trying? It’s certainly not documented anywhere, see for example Gnome discussion from 2015 https://wiki.gnome.org/Initiatives/Wayland/PrimarySelection, which is long out of date but still pretty much the top search result for this. (But preserves the lovely gnomeism of describing something they don’t want to do as an “easter egg”. Hey guys, document things properly and they aren’t easter eggs, but that’s probably too difficult.)

Currently looking forward to whatever replaces wayland. Back in the late 200x-es we got a nice view of what the future of linux desktops might be like in Compiz, and then spent the next decade or so going backwards while they chased single user Macbook functionality.
———————
Middle click paste from a primary buffer is a core function for me and a total deal-breaker if not available.
———————
There are way too many things that X can do that Wayland either cannot yet do reliably, or even at all. Therefore Wayland is just not an option for me until that is fixed.
———————
Over now to my favorite quote from elsewhere:

You failed to read the fine print at the bottom of all the wayland promises over the past 12 years:

It will improve your performance. Next year. Or the year after that. Or maybe the year after that. If you have the right hardware. And the right desktop. On certain tasks with certain apps. Maybe. Depends on the alignment of the stars and the moon, and if Jupiter is in the 2nd house”.

Can metastases be cured?

Because this is what systemd and Wayland are.

Windows 11 has metastatic cancer. Apple’s shit only runs on Apple’s shit, which is expensive without being worth it. Linux has metastatic cancer. FreeBSD, NetBSD and OpenBSD don’t support most hardware on most modern laptops.