I had my suspicions that Fedora Linux is maintained by a bunch of morons who misinterpret or interpret too strictly Red Hat’s requirements, but now I have the proof that they’re literally retarded. In the process of rejecting a “Fedora AI Developer Desktop,” they also rejected the adoption of an LTS kernel!

For those who are too innocent (that’s to say that they don’t know shit), Fedora Linux is the only non-rolling distro that does not keep using the same kernel, not even during the first 6 months of a release, and not even if they happen to release with an LTS kernel! Every time a new major kernel version is released, Fedora adopts it after a certain delay. Heck, even Arch Linux offers a linux-lts kernel!

I learned about that via LWN.net, who nonetheless couldn’t be bothered to notice the kernel issue: Friction in Fedora over AI developer desktop initiative. The full story requires a subscription, but the front page includes this portion of the news:

A push by Red Hat employees to create a Fedora “AI Developer Desktop” with support for out-of-tree kernel drivers and AI toolkits has been met with objections from some long-time members of the Fedora community. After more than a month of sometimes heated discussion, the Fedora Council had voted to approve the initiative; however, a last-minute change to vote against the proposal by council member Justin Wheeler has (at least temporarily) sent it back to the drawing board.

Maintaining out-of-tree kernel drivers is indeed delicate, especially when this is about NVIDIA, but the discussion was literally absurd. Some excerpts follow.

First, from Gordon Messmer’s description of the Fedora AI Developer Desktop Objective:

The Fedora AI Developer Desktop Initiative aims to build a thriving community around AI technologies by focusing on three key areas: equipping developers with the necessary platforms, libraries, and frameworks; ensuring users experience painless deployment and usage of AI applications; and establishing a space to showcase the work being done on Fedora, connecting developers with a wider audience.

…

Non-goals:

The system image will not be pre-configured with applications that inspect or monitor how users interact with the system or otherwise place user privacy at risk.

Tools and applications included in the AI Desktop will not be pre-configured to connect to remote AI services.

AI tools will not be added to Fedora’s existing system images, Editions, etc, by the AI Desktop initiative.

…

Deliverables

A number of changes would produce an operating system image that would improve Fedora as a platform for AI software. This work may include:

1. Build an LTS kernel to deliver the benefit of a stable release process consistently, across the release.

2. Build and sign the NVIDIA out-of-tree kmod, OpenRM (until the Nova driver is ready), to eliminate the need for signing keys on the device where the keys are used, to provide a standard best practice build-test-deploy sequence, and to enable an Atomic system to support NVIDIA hardware.

3. Publish an Atomic system image focused on support for accelerated AI workloads, as a Fedora Spin.

3.1 Publish a variant of that Atomic system image with CUDA runtime support, as a Fedora Remix.

4. Publish an Atomic system that supports the CUDA toolkit. (If Fedora cannot distribute this image due to license or policy issues that we can’t resolve, I’d like to ask NVIDIA if they would publish the image we build.) This layered image will be a Fedora Remix.

5. Include user-friendly tools common to various AI back-ends, such as Goose CLI and Podman Desktop.

6. Provide a contribution guide and invite community developers to directly improve the image rather than writing post-installation configuration guides.

7. Provide a space for AI software developers to showcase their work, provide short quick-start guides, and help users find applications that solve their problems and projects that they can contribute to.

8. Publish optimized container images for machine learning applications, aimed at AI developers.

So far, so good.

Note that Gordon Messmer himself, when answering to the question, “Would this LTS be specifically for AI Developer Desktop?”, supported a general availability of an LTS kernel:

I expect a stable kernel to benefit people in many audiences:

  • In the general case, I often hear people complain about hardware support regressions after kernel feature upgrades. I myself was unable to use Fedora kernels 6.15+ on Fedora 42 because my trackpad wasn’t usable on newer kernels. A stable kernel ensures security patches to users affected by regressions.
  • In the case of systems that use out-of-tree modules (including NVIDIA’s in the context of this proposal, but also things like VirtualBox or ZFS), a stable kernel provides support for the use of third-party kernel modules while they are ported to new kernel interfaces.

An additional kernel would present greater QE overhead. I don’t want to deny that. But I do think that the testing process around Fedora kernels today has serious flaws. The rolling kernel necessitates testing days mid-release, which does not align well with the concept of a stable release. But more importantly, even if users participate in testing days and report regressions, there just isn’t any realistic alternative to shipping the new release series as an update. By the time Fedora prepares a new kernel release series and organizes a test day, support for the previous release series has ended upstream. We can take bug reports regarding regressions, but there isn’t much we can do with them.

But then, all hell broke loose!

The most vocal opponent was Neal Gompa, a very busy, high-profile packager who sits on the Fedora Engineering Steering Committee and on the AlmaLinux Engineering Steering Committee and contributes to Asahi Linux, CentOS SIGs, openSUSE, Mageia, and whatnot:

I don’t think this is a useful change. Firstly, LTS kernels are almost exclusively used for development of components that aren’t integrated or supported upstream in any meaningful way. Not to mention, the complexity of supporting alternative kernels in Fedora software and update infrastructure is significant. And finally, it screws over our ability to coherently identify hardware support.

…

Finally, there is another issue changing our policy to support and endorse out of tree kmods in Fedora as KMPs: the likelihood our user’s systems will be considered tainted and ineligible for support from upstream kernel developers goes up significantly. Kernel developers prefer Fedora and Fedora users over others because of the existing state of things.

In other words, Fedora users are guinea pigs.

Further defending the status quo:

One of the problems Fedora has right now is that its primary sponsor and major employer of kernel hackers does not generally allocate their contributors to participate in Fedora. They don’t review Fedora bugs, they don’t assist users, they don’t help much with the Fedora kernel in general. It is telling that ARK and the kernel package actually define Fedora as the special case, rather than RHEL being the special case as the subset downstream. ☹️

As a counterpoint, Btrfs in Fedora is well-supported because the Btrfs SIG has upstream Btrfs developers as part of the team and actively does work on that. When reports come in, they are triaged and responded to either within Fedora or upstream. The fact that we ship the latest stable kernels means that those fixes are delivered quickly too.

Gordon Messmer:

A stable kernel package would continue to deliver security patches and serious bug fixes to users who can’t rebase to a new kernel series because of regressions or software support.

…

RHEL is a product. Red Hat determines the intended use cases to fit the markets they want to participate in. They develop the product to support those use cases.

Fedora is not a product. The software we provide is unsupported, which is to say that we are not accountable to users. If the thing we build the publish works, they may use it, and if it doesn’t work, they can maybe participate and try to fix it or they can use something else. But there’s no mechanism for them to hold us accountable otherwise. Unlike Red Hat’s model, in which they are offering continued maintenance of components independent of upstream, Fedora does significantly less in-distro development, and is mostly doing distribution.

…

The purpose of a stable release system is that it allows groups to collaborate asynchronously. It allows the developers of a product to publish a release series to get a feature set in to the hands of interested users. The groups of users for whom that series works can upgrade immediately, and the groups for whom it doesn’t work can work on porting and bug fixing until it does. The rolling kernel timelines don’t overlap enough for that to be realistic. Regressions and porting cannot reasonably be addressed within the overlap. Users would suffer regressions, even if more developers were actively working on bug reports against the Fedora kernel. In other words, even if developers were addressing bug reports, we would need more overlap in maintenance windows AND we’d need a mechanism by which users could upgrade from one kernel release to the next on their own schedule, rather than shipping one rolling kernel package.

…which is exactly why a stable kernel that spans a release would benefit many users. I’m actually quite surprised to see anyone argue simultaneously that there are not enough developer resources assigned to the rolling kernel release and that the stable kernel isn’t useful or desirable. Those seem like contradictory points of view, to me. The latter is the solution to the former.

Neal Gompa:

Firstly, there’s no such thing as a “stable” kernel by your definition (frozen API/ABI, etc.). Even the longterm kernels do not provide that guarantee, which is why Red Hat had to do extra work in the RHEL kernel for it, and they stopped doing that in RHEL 9.

…

Well, I disagree here, because Red Hat is notably the only one of the big three commercial Linux vendors to not do this. Both SUSE and Canonical have their Linux kernel teams actively engaged and supporting their community platforms (openSUSE Tumbleweed and Ubuntu STS, respectively). I have personally experienced getting assistance and upstream kernel fixes delivered in both platforms simply by filing bug reports in their community spaces. This is something Red Hat currently fails at and it’s really baffling.

By definition, Fedora being the upstream to their products means that it’s better for Red Hat supply resources there to improve the quality of things there. It benefits RHEL too because their kernel regularly rebases whole subsystems for every minor release. In this sense, even the RHEL kernel “rolls”.

So there is no project or product rationale here that justifies the “longterm” kernel in Fedora. It doesn’t improve stability, it doesn’t improve quality, it just creates a “holding pattern” that doesn’t actually fix the problem.

I’ve never seen such an asshole in my entire life!

Bottom line: Fedora will never have an LTS kernel. Never ever. And that’s because Neal Gompa doesn’t want it to happen. Not Red Hat’s CTO, but Neal Gompa.

Side notes

Fedora being upstream for RHEL is a myth. There’s another layer between Fedora and RHEL, and that’s CentOS Stream. RHEL releases rarely, but when it does, it pulls from CentOS Stream.

Minor releases import packages from CentOS Stream. Major releases are also pulled from CentOS Stream, but this is when packages from Fedora enter CentOS Stream.

However, most packages in most Fedora releases only live in Fedora. They never reach CentOS Stream and even less RHEL.

Furthermore, most packages only live in Fedora because since version 10, RHEL is transitioning to providing desktop applications via Flatpak. EPEL doesn’t offer as many packages as before (EPEL 8 and 9 did offer KDE and XFCE, but EPEL 10 only offers KDE).

The idea is that Fedora has its own purpose and should have more leeway with regard to RHEL. Neal Gompa is wrong: Fedora has its own independent value and purpose, not just as RHEL fodder.

On the other hand, while Red Hat Enterprise Linux for Workstations and SUSE Linux Enterprise Desktop can be used on desktops and business workstations, there is no commercial Linux vendor to specifically target the corporate desktop. There are hundreds of millions of corporate Windows desktops (and laptops) that could be converted to Linux, but nobody bothers.

Furthermore, while Ubuntu is very popular on individual desktops (and laptops), and there are some Ubuntu-certified devices, Canonical also targets other areas of business Linux: servers, containers, AI, specialized kernels, and so on.

Literally nobody attempts to carve into Microsoft’s market share in the corporate environment.

In this context, Fedora, which “is not a product,” is only a testbed for some kernel developers and a trap for a lot of guinea pigs.