Kernelstub doesn't work with multiple kernels

[ayoungethan]

Distribution (run cat /etc/os-release):

NAME="Pop!_OS"
VERSION="19.04"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Pop!_OS 19.04"
VERSION_ID="19.04"
HOME_URL="https://system76.com/pop"
SUPPORT_URL="http://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=disco
UBUNTU_CODENAME=disco

Related Application and/or Package Version (run apt policy $PACKAGE NAME):

kernelstub:
Installed: 3.1.0153936005819.0464d0caf
Candidate: 3.1.0153936005819.0464d0caf
Version table:
*** 3.1.0153936005819.04~64d0caf 1001
1001 http://ppa.launchpad.net/system76/pop/ubuntu disco/main amd64 Packages
1001 http://ppa.launchpad.net/system76/pop/ubuntu disco/main i386 Packages
100 /var/lib/dpkg/status

Issue/Bug Description:

Kernelstub does not handle more than one kernel installed at once. In my case, linux-lowlatency kernel. Instead, kernelstub treats the previously installed kernel as the old/backup kernel. This means that no reliable boot menu option for selecting which kernel to load at boot time can exist.

The whole point of installing more than one kernel is to be able to switch between them freely. Kernelstub does not currently allow this.

Steps to reproduce (if you know):
Install a second kernel line, such as linux-lowlatency. System will automatically boot into the new kernel, and the user won't have access to the linux-generic kernel until the linux-generic kernel updates, at which point, system will automatically boot into the linux-generic kernel until the linux-lowlatency kernel updates.

Expected behavior:
Kernelstub should create and maintain menus for kernel lines other than linux-generic as they are installed.

A clear policy for previous kernel versions should exist. Probably the safest is to allow for at least one previous kernel version for every kernel line in the boot menu. In my case, this would be:
linux-generic (current)
linux-generic (previous)
linux-lowlatency (current)
linux-lowlatency (previous)

Other Notes:
Current manual workaround:

1) Install the lowlatency kernel with apt/dpkg:

sudo apt install linux-image-lowlatency linux-tools-lowlatency

2) After you install or update the lowlatency kernel you need to copy it and the initrd for that kernel to the ESP partition (/boot/efi)

cp /boot/initrd.img*lowlatency /boot/efi/EFI/Pop_OS-627944f7-4f7b-4f47-80f7-2ca16e1d04a1/initrd-lowlatency.img

cp /boot/vmlinuz*lowlatency /boot/efi/EFI/Pop_OS-627944f7-4f7b-4f47-80f7-2ca16e1d04a1/vmlinuz-lowlatency.efi

3) Copy the loader entries created by kernelstub:

sudo cp /boot/efi/loader/entries/Pop_OS-current.conf /boot/efi/loader/entries/Pop_OS-lowlatency.conf

4) Edit the new entry so you see it as lowlatency:

sudo nano /boot/efi/loader/entries/Pop_OS-lowlatency.conf
edit "title" from "Pop!_OS" to "Pop!_OS-lowlatency"

5) Update the loader entry that you already created to point to the kernel and initramfs image you copied over (/boot/efi/loader/entries)

title Pop!_OS-lowlatency
linux /EFI/Pop_OS-627944f7-4f7b-4f47-80f7-2ca16e1d04a1/vmlinuz-lowlatency.efi
initrd /EFI/Pop_OS-627944f7-4f7b-4f47-80f7-2ca16e1d04a1/initrd-lowlatency.img

6) Edit the loader.conf file to add a timeout on boot so you can choose the kernel you want:

sudo bash -c "echo timeout 5 >> /boot/efi/loader/loader.conf"

[ayoungethan]

I notice that in /boot/efi/EFI/Pop_OS*/ the images copied from the /boot directory could benefit from a standardized heuristic naming scheme. Currently, for example,
whatever the last-installed kernel gets copied to
admin:///boot/efi/EFI/Pop_OS-b55ea11c-b440-4fb2-874f-86a5768fd782/initrd.img and
admin:///boot/efi/EFI/Pop_OS-b55ea11c-b440-4fb2-874f-86a5768fd782/vmlinuz.efi

The previous kernel initrd gets copied from initrd.img to initrd.img-previous, whereas the vmlinuz.efi gets copied to vmlinuz-previous.efi

This could be standardized to append -previous. (vmlinuz-previous.efi and initrd-previous.img)
Then this same idea could be applied to a heuristic process with kernel names, reading the version number and the kernel description to understand how to copy and rename from /boot to the EFI partition.

In my case, if/when I install or upgrade -generic and -lowlatency kernels, a filename parser could catch the -generic and the -lowlatency tags on the filenames and append those to the EFI partition files, so instead of a "vanilla" initrd.img and vmlinuz.efi I would have

1. Generic: initrd-generic.img and vmlinuz-generic.efi and
2. Lowlatency: initrd-lowlatency.img and vmlinuz-lowlatency.efi

This would depend on the kernel filename and versioning being consistent.

The -previous entries can always refer to the previous -generic kernel version, as I understand their use case is simply as a means of recovering from a fatal bug or regression in a newer kernel. OR the bootloader can use the heuristic to maintain parallel -previous entries of different kernel lines (e.g., initrd-generic-previous.img and initrd-lowlatency-previous.img). OR a script could parse both version and install date and keep the 3rd most recent version/install (regardless of which kernel line it comes from) as the -previous entry.

Another workaround for this problem for people who need to only run a single kernel is to allow the System76 Driver to depend on any Pop!_OS kernel, so the -generic kernel needn't be installed, for example, if someone plans primarily to run -lowlatency.

[krekas]

Having same issue

[daliborfilus]

I wanted to try low latency kernel for SteamVR to see if stuttering will be lower with it, but I couldn't because of this. Will try the workaround, but it's definitely an issue which needs resolving, especially if the pop packages prevent us from switching the kernel completely.

[cloudishBenne]

I finally created a fully working manager custom-kernel, which operates on top of, but separate from, kernelstub. I've used earlier prototypes of this program for 2-3 years, and two of my friends also use it without any issues. It took me a long time to polish it into a 1.0.0 release, but I finally pushed it!

If @isantop is interested, I'd like to discuss the methods I used. Ideally, there should be no 2.0.0 of custom-kernel, as I'd prefer to merge it as a feature into kernelstub, provided there are no massive downsides or shortcomings in my code.